John James Jacoby
Andy P
Paul Gibbs
Boone B. Gorges
Andrew Nacin
Since the inception of WordPress’s custom post type API, the BuddyPress core team has been bouncing ideas back and forth on how we could possibly take advantage of all the goodness it has to offer. With bbPress recently making the move last year, and BuddyPress taking advantage of that in the coming version, we’re slowly and carefully inching towards trying it out.
A few things to consider:
- The scalability concerns of BuddyPress on single-site WordPress become apparent much more quickly when everything is in the same set of WordPress database tables
- Even something as straightforward as bbPress had to pull serious strings to use custom post types. Components like Groups, Friends, and Activity will be exponentially more complex.
- By moving to custom post types, we inherit TONS of awesome features we don’t have in BuddyPress today: Trash, Import/Export, post_status, roles/caps, taxonomies, etc…
- Migration is a pain, and writing scripts to transition each component over safely is hugely important.
- Custom post types take much of the weight off the BuddyPress core team. It lowers the barrier of entry to get familiar with BuddyPress, and hopefully will increase both WordPress and BuddyPress core contributions.
- Some things just aren’t meant to be custom post types. Things like user-to-user relationships, profile fields, etc… The schema for post types doesn’t really make any sense here; does it make sense to move components over just for the sake of doing it?
Any feedback is great feedback. We really want to get as much as we can get from everyone in the WordPress community, to get a sense on the direction you want us to take for future versions. We think we’re on track, but we won’t know unless we ask. 
Thanks in advance for your time and thoughts.
ericslangley, 
Vittorio, 
kim, 
Deepak, 
Arturo, and 1 other are discussing. 
David Bisset 3:16 pm on March 5, 2012 Permalink |
This isn’t going to be insanely helpful, but coming from someone who has used CPTs more in the past 2 months than I ever have since they came out – I can’t help but say moving to use CPTs is the future. Sounds like though there is going to be another massive wave of rebuilding and recoding BP plugins – from which i’m sure we’ll be hearing moans, groans, and debates about.
I didn’t understand your first comment though John… about scalability. I assume w/ CPTs BuddyPress can scale better/faster/stronger (Daft Punk)? Will there be better caching, etc.?
Would CPTs then be a BuddyPress 2.x thing?
Stas Sușcov 3:35 pm on March 5, 2012 Permalink |
Agree with @dimensionmedia
Imho, the way we should put the problem regarding use of CPT in BuddyPress is not if this technology is ready/scalable, but how well it integrates into the whole codebase as a framework component.
Since CPT are part of WordPress I don’t see why those can’t be fixed if there will be scalability problems.
On the other side, using CPT means we will inherit a great set of already existing and stable “benefits” (features, testing tools, plugins, developers, docs).
Imho, going with CPT is the DRY way of improving a lot of things BuddyPress lacks at this moment.
John James Jacoby 4:26 pm on March 5, 2012 Permalink |
The greater majority of changesets in 1.5 and 1.6 were to encapsulate things into callable functions with API’s so that if/when we want to modify the schema, plugins that are doing things as intended won’t break.
Example: If WordPress core changes the API for WP_Query, ideally get_posts() still works the same as it did before. That’s happening now in WordPress 3.4; it’s the same goal we’d strive for, too.
John James Jacoby 5:41 pm on March 5, 2012 Permalink |
Scalability in terms of users to data created. Stuffing activity streams with forums and blogs and groups all into 1 set of database tables slows everything down, instead of having a custom schema purpose built to do the job efficiently for each component.
Imagine running WordPress.com, 35 million sites, on 1 set of tables. You can’t. I know that isn’t a typical installation, but it’s a potential goal of a social network. Blogs traditionally have 1 user, social networks hopefully have more than 1.
Stas Sușcov 6:28 pm on March 5, 2012 Permalink
I agree with the situations you described, but for average BuddyPress user, as it is for average WordPress user, scaling will mean dozens, hundreds of users.
For the rest of the monsters, there will always exist caching technologies.
If I remember WordPress at its early stage, I think #1 task for them was to make it appealing for end-users, not for enterprise.
Stas Sușcov 3:36 pm on March 5, 2012 Permalink |
Why this is tagged pizza btw?
John James Jacoby 4:15 pm on March 5, 2012 Permalink |
Because Pizza is delicious.
Remkus 3:46 pm on March 5, 2012 Permalink |
IF moving forward is the main concern then it would seem to me that moving over to CPTs is the way to go. I know for a fact that this will make a lot of devs currently not looking into BuddyPress – because they’ve got to learn how it works and they don’t have the time to do it – will find it a lot easier to move once BuddyPress is built on top of CPT’s.
John James Jacoby 4:13 pm on March 5, 2012 Permalink |
My counter to this, is moving bbPress over to custom post types has not dramatically increased developer contributions to the project. I agree with the hope that it’s a long-term investment that will pay-off, but time will tell.
Remkus 4:25 pm on March 5, 2012 Permalink
I know a lot more devs are playing with bbPress when the wouldn’t touch it before..
John James Jacoby 4:36 pm on March 5, 2012 Permalink
If that’s true, I wish they’d stop by and help out on trac a bit more.
Mika Epstein (Ipstenu) 5:13 pm on March 5, 2012 Permalink
I would be willing to put money on the drop-out from bbPress being more because of the, shall we say, narky attitude in the forums for so long. Burning out + massive shift + no apparent leadership. Lead to in-group sniping. I know that’s certainly why I use bbPress, but don’t contribute to the forums/trac (much… I have tossed out things at JJJ).
Marcus Sykes 5:22 pm on March 5, 2012 Permalink
speaking personally, i think it’s a time-issue. various projects which can warrant bbpress that have already been started and we can’t jump ship until time permits. e.g. had to configure faultpress recently, when I wish I could now be using/customizing bbp
).
scribu 4:05 pm on March 5, 2012 Permalink |
I think you should move to custom post types only the components that make sense. Don’t try to shoehorn everything in just to avoid creating a few custom tables.
Chris Clayton 6:22 am on March 13, 2012 Permalink |
Was going to say exactly what Scribu just said, move the things that make sense. Post types are types of content, correct? Members and Groups aren’t really content… but forum posts and Activities are content.
I personally think that better post type integration with buddypress (such as boones ticket – http://buddypress.trac.wordpress.org/ticket/3460) should be more of a priority than moving over the current stuff we have.
Boone Gorges 4:13 pm on March 5, 2012 Permalink |
I think what John means by ‘scalability’ is that moving to CPTs for all data types means moving more and more data into the wp_posts and wp_postmeta tables. For example, on a moderately large site, you might have 10,000 users, 500 groups, 10,000 forum posts, and 500,000 activity items (in addition to other miscellanea). Putting all this data into a single table means that the table will get into the millions, tens of millions, hundreds of millions of rows more quickly than if the pain were spread between different tables. (The problem is multiplied by about an order of magnitude for postmeta.)
Large table size itself is not necessarily a bad thing (MySQL is optimized for that sort of thing). But keep in mind that the size issues become more pressing when we have to do a lot of tricks to fit BP data types into the WP post schema. Take something like an activity item. It currently has metadata like item_id and secondary_id. As a CPT, that data would presumably be stored in postmeta or in a taxonomy. That means costly meta_query or tax_query parameters with every bp_has_activities() call, instead of optimized WHERE clauses. It gets even more complex with other components: how do you store the recipients of a message, for instance, or data about friendships?
As for the argument that CPTs will make BP more appealing or easier to approach for developers, I have become convinced that this isn’t really the case. Take friendships. Right now you query for friends with
bp_has_friends( array( 'user_id' => $user_id ) )With CPTs, the details would depend on how we implemented the schema, but the underlying query might be something like:
$friends = new WP_Query( array('post_type' => 'bp_friends',
'tax_query' => array(
'relation' => 'OR',
array(
'taxonomy' => 'bp_friend_initiator',
'field' => 'slug',
'terms' => array( $user_id )
),
array(
'taxonomy' => 'bp_friend_friend',
'field' => 'slug',
'terms' => array( $user_id )
)
)
) )
Is that really easier or more approachable for WP devs?
The biggest benefit of moving some components to CPTs, IMO, is inheriting WP features and optimization. I’m thinking specifically of post-specific caching mechanisms.
As scribu said – it will make sense for some components (forums via bbPress, maybe groups, maybe messages) but probably not for others.
John James Jacoby 4:23 pm on March 5, 2012 Permalink |
Exactly.
These are problems we can solve, but are they problems that currently need to be solved? I don’t think so, not yet anyways.
Caching would be a huge win, but we get some of that already by way of using the WPDB class and caching our direct queries. I think we have some cache invalidation issues to work out still, but moving to CPT doesn’t make those go away, either.
Remkus 4:29 pm on March 5, 2012 Permalink |
I agree with you in this example, that is doesn’t make it easier persé, but it is what they know and know how to play around with. That’s all I am saying
croixhaug 6:38 pm on March 5, 2012 Permalink
I agree with boone, the CPT querying isn’t much easier/more appealing to approach, at least not enough to entice developers. I personally fall into the non-contributing dev. I work on a few BuddyPress projects but find myself “waiting in the wings” when it comes to helping out with general BP development. It’s mostly not knowing where to start. Changing to CPTs might help a bit, but realistically that’s not what’s holding me back.
Marcus Sykes 4:53 pm on March 5, 2012 Permalink |
As someone that has spent the past 6-12 months doing just this to the Events Manager plugin, I feel I have something relevant to say
I agree that some things could go in the posts table, but certainly not others.
For example, I’d very strongly recommend you not add something like user activity there, or any other table that inserts transactional records of this nature, as you expect, if your site gets popular you’ll get a very bloated DB.
Moreover, aside from what Boone says, I have my doubts as to how fast lookups will be when you’re joining tables left and right, especially for simple stuff like group memberships etc. I’m actually going to code in a way to use both in my plugin (I use a mirrored index table for searches, remenants of older versions, but has it’s uses), so I’m quite interested to see what happens when I benchmark the two on a 1-10k events site.
Yes, there’s an argument to the fact that WP should be the ones optimising the retreival side of things, but in my case, I can’t think of many people that would like to see user booking information on the same table they store their site content.
I should look and see what you guys did on bbPress, I had do do some dodgy stuff myself
John James Jacoby 4:56 pm on March 5, 2012 Permalink |
bbPress did some pretty straight forward things with post meta and template loading, but it took 30k lines of code and a year to get there.
Marcus Sykes 5:13 pm on March 5, 2012 Permalink
same here, will have to check that out for sure then, template loading was what I was referring to as dodgy
John James Jacoby 5:36 pm on March 5, 2012 Permalink
Checkout braches/plugin. There are 3 files in /bbp-includes with a bunch of awesome logic that’s meant to be reusable. Output buffers for template parts, query parsing, $post overloading, etc…
Ryan Hellyer 8:34 pm on March 5, 2012 Permalink
Hmmm, I had no idea you had to jump through so many hoops just to get that stuff working. Thanks for putting in all the hard yards
Deryk Wenaus 5:06 pm on March 5, 2012 Permalink |
It seems the general consensus is to consider moving over components that make sense, and leave the others as they are. That makes total sense to me. And is more of an evolutionary, less jarring approach.
As for relationships, perhaps the BP team can be the ones to finally get post and user relationships into core (http://core.trac.wordpress.org/ticket/14513). What a blessing that would be for the entire WP developer community!
My concern is less about speed and scalability and more about codebase simplicity. Right now the BP core is a bit of a beast, and has a steep learning curve. After a transition to CPTs are made for some content, I’m guessing the code base would be simpler and easier to comprehend (putting aside the code needed for the transition), would I be wrong about this?
Boone Gorges 5:17 pm on March 5, 2012 Permalink |
> I’m guessing the code base would be simpler and easier to comprehend (putting aside the code needed for the transition)
Depends on the implementation. Obviously most of the changes to the code would be in the database classes – what you call the ‘translation’ – which, depending on your point of view, may make that code easier or harder to read. (Do you prefer reading functions that concatenate SQL, or those that concatenate WP_Query args?
) There is also the question of template tags. We currently load the results of the DB query into “template global” – $groups_template, $activity_template, etc – and then our template tags reference values in those globals. We could conceivably leave this the way it is, which would maximize backward compatibility and wouldn’t change any of the template tag code. Alternatively, we could wipe out the structure of these globals, and turn the template tags into wrappers for the WP versions: so
bp_get_group_name()is a wrapper forget_the_title(), etc. This may make code easier to read from the developer’s point of view, but would probably break a fair number of plugins and hacks that modify the template globals.John James Jacoby 5:31 pm on March 5, 2012 Permalink
Transitioning to magic methods and privatizing variables into magic getters and setters gives us a mechanism for backwards compatibility we didn’t have at our disposal with PHP4.
I think we could transition our globals into private static instances where it’s needed, without too much pain. I fear we’re stuck with the one true $bp global forever though, since we’ve always pushed for it to be the container for anything BuddyPress related that needed to be available in global scope.
Marcy Capron 5:47 pm on March 5, 2012 Permalink |
Okay so I am coming at this from the perspective of building really weird things with WP/BP. Not the typical use, probably. Constant hacking at to build “minimum viable products” for startup founders.
Our dev lead basically said that he gets the usefulness of BP such as messages and some of the other social components but that it feels so bloated (and not easy enough to turn any one part off…. so much code loaded) that he gave up (right before 1.5 I think) …. even though I begged him not to reinvent the wheel with things he needs. Much of what he likes from BP he’s sorta of re-built himself using source from BP, other WP plugins and his own library… to better suit the a la carte nature of our work.
I, personally, am finding newer BP to be much more of a breeze (working on two small projects on my own) and the way bbPress has changed has made my life WAY easier.
So if BP follows bbPress suit I’d be pleased. I think in general compartmentalizing features into things that can be used/not used/turned off with CPTs and such would be great. Make it more of a puzzle piece scenario where greater things can be built when chunks are combined, but get rid of how contingent each piece is on each other individually.
p.s. unrelated – the comment system here on this wp.com blog is pretty magical ajax stuff, huh! fun to watch it load in new stuff, live, while I’m typing this…
Mika Epstein (Ipstenu) 8:11 pm on March 5, 2012 Permalink |
I realized I didn’t reply usefully! I blame … J-trip. Anyway, I think ‘where applicable’ it should be CPTs. That makes content easier to manage, as well as application of plugins (Akismet for example). bbPress went from ‘this thing that mostly works, if you’re savvy and have a sledgehammer’ to ‘Boom goes the dynamite! DONE!’ Groups look like the sort of thing CPTs were made for.
The only thing I ‘dislike’ about bbPress is how many menus there are on the back end. Forums, Topics and Replies. Then there’s ‘recount’ in tools, and ‘Forums’ in settings. BuddyPress has a nice consolidation, and I wouldn’t like to lose that.
Ryan Hellyer 8:30 pm on March 5, 2012 Permalink |
I don’t put the bbPress menu system down to custom-post types necessarily. You can move/create menus however you want; it just requires jumping through a few hoops to achieve it. You aren’t necessarily stuck to a certain setup simply because the custom post-type defaults to something.
Marcus Sykes 8:48 pm on March 5, 2012 Permalink |
Good point Mika, BP will have to consider the auto-menus too. I had this problem with events/locations and had to make a few adjustments to merge them into one menu (started off with multiple, just didn’t look good). More and more plugins will run into this once everything becomes a cpt so I bet something will be done about it.
Joachim Kudish 8:27 pm on March 5, 2012 Permalink |
I’m very much with others on this one. I do think that a CPT conversion would be beneficial for some but not all components. It would make them easier to work with it and the added benefits (admin interface, trash, statuses, etc…) are definitely worth it. At the same time, as others mentioned, it doesn’t really make sense to have profile_fields stored as a CPT (maybe they could even just be a serialized array in the options table?).
Anyways, just my 2 cents. I already use BP when needed/possible for projects and having CPTs in there would only be an added bonus/feature and bring it close to WP core functionality.
Foxly 7:11 pm on March 7, 2012 Permalink |
As somebody building a 1 million+ user BuddyPress site, I strongly recommend you avoid custom posts altogether and jump straight to a custom DBO (database object) class.
1) Using CPT’s will only work for about 1/3 of the storage needs in BuddyPress, but will *double* the storage interface complexity. Developers working with your code would then need to learn how to use two different data interface paradigms, making work slower and more error prone.
2) Using CPT’s will not significantly improve the speed of BuddyPress. The speed problems in BuddyPress are being caused by your data model and your caching implementation.
3) Switching to CPT’s probably won’t help you attract more developers. Because of what it does, BuddyPress is a huge, complicated platform, and requires expert-level coding skills to make meaningful dev team contributions.
We’ve spent thousands, and thousands, and thousands of hours solving the problems you’re currently trying to deal with.
We’ve quietly built solutions for pretty much all of them.
We’d be happy to teach you how to use them and help you migrate your code over. And yes, *our stuff will work* on the fragmented db setup buddypress.org runs on.
Although its possible to do pretty much anything with [insert list of random WordPress features], doing it for a couple of million users is a completely different story. Our solutions are designed to work at the *million user* scale.
Caching: http://code.google.com/p/buddypress-media/source/browse/bp_media/trunk/core/cache/class.memory.cache.php
Why use WP’s incredibly limited built-in caching when you can use BPM_cache to interface with APC and Memcached *simultaneously*, along with full namespacing support.
Database Interface: http://code.google.com/p/buddypress-media/source/browse/#svn%2Fbp_media%2Ftrunk%2Fcore%2Fdatabase
The most powerful WP/BP database class ever written. Fully object-oriented, to the point where developers need little if any knowledge of SQL. Immune to all known SQL-injection attacks, with strong-typing and full typecasting between PHP and SQL data types. Supports transactions. Full unit tests with 100% code coverage.
Docs: http://code.google.com/p/buddypress-media/wiki/DOCS_BPM_db_top
Data storage classes:
Imagine if building a massive data silo was as simple as defining an array. Everything else, from creating the DB table to implementing a writeback cache with dictionary support to data access functions was automatically done for you.
http://code.google.com/p/buddypress-media/source/browse/bp_media/trunk/core/page_modules/class.page.module.manager.php
The base class:
http://code.google.com/p/buddypress-media/source/browse/bp_media/trunk/core/base_classes/abstract.class.module.manager.php
And full docs:
http://code.google.com/p/buddypress-media/wiki/DOCS_BPM_cache_smallCache
BuddyPress abstraction API:
http://code.google.com/p/buddypress-media/source/browse/bp_media/trunk/core/abstraction/class.bp.abstraction.php
Just by 0b10 cents.
^F^
Warzan 10:28 am on March 8, 2012 Permalink |
I started reading this thinking your getting very cocky these days @foxly ;o) But to be fair, you do have a point.
We run probably one of the larger and more busy implementations, and for us speed has always been about caching and cloudfront. We opted for W3TC but never got it to play well with BP – Heck i even see a ticket about caching issues with forums allocated to 1.6 so hopefully it will be resolved.
CPT while nice for factoring content are probably best left for that ‘content’ – the social side is 99% noise and probably is best served within its own optimised database structure.
Currently I don’t believe CPT have matured enough for them to be considered as the basis of ‘Apps’ UMBRACO has a fantastic framework under the hood and is already where wordpress is rapidly heading, but its back end allows you more flexibility in the presentation of its ‘post (data) types’ making it easier for the user to distinguish between content, verses default data, verses collected data etc.
Back to Caching
It has to be said though that while wordpress is pretty scalable these days, buddypress is not so mature in that respect.
We have struggled with the design/development decisions taken on this project such as the dumping of default themes etc to the point where upgrades become nightmares. (We have had to fire up a project of its own to basically start from scratch with the existing database)
I wonder if the prospect of ‘shiny new features’ sometimes takes a little precedence over what a social network is supposed to do.
In case anyone in the development team has forgot or need a little reminder let me give you a little list of what most installations require:
1) We want it to be simple for users to sign up
2) We want users to take pride in their profiles and be interested in visiting other peoples profiles – profiles should reflect the user and what their doing and their personality
3) It should be easy for users to leave comments and messages for each other on each others profiles, and use profile pages as a simple messaging platform
4) Users should be able to share the stuff that interests them, by dropping it onto their profiles or each others – People share photos, videos, events, links and memories
5) People should be able to highlight the stuff in the network so that their friends can go check it out.
6) People should be able to find new friends and connections on the network easily and should be continuously prompted to make new friends.
7) People should have three home pages 1) The main blog homepage 2) The main Activity Stream 3) Their Own Profile – From these spots they should get a good overview of everything.
8) People should be able to invite their friends into the network by email, and when posting an update optionally inform their friends by email)
8) It needs to sit on top of facebook and twitter because whatever way you swing it THEY ARE the social networks, we are just little but highly functional groups on the outside.
9) It should scale without having to be a rocket scientist to at least handle 5M page views per month on a reasonable dedicated box.
10) Users should be able to protect their privacy to a reasonable level and control what people see about them.
You have to consider of the above:
1) What does buddypress currently do well? (I’m hard pressed to say it does any of the above in any spectacular fashion)
2) What of the above should be core functionality that people can count on? (Again we have a social network platform that doesn’t facilitate people sharing photos – can anyone see how ridiculous that is?) – Ohh and before the usual we have plugins response is deployed – plugins are one of the biggest security threats and weaknesses of the whole BP environment, you cannot be sure what you are installing, and if you think a plugin is good, then shame on you for not bringing it into the core and maintaining it properly)
3) What has went so wrong that in 3 years we are rally no further along in any of the above, and are now debating how cool custom post types are? I fail to see how custom post types will help solve much of the above list.
As an inside outsider (if you catch my drift) I just see the BP community fragmenting more and more, and people just not working together.
We can put it down to developers not being paid, but perhaps its just about the rewards.
Jeff was working on privacy then walked off citing the rewards are just not there, why did he not get more support from core? Is the privacy of data not considered a core component?
Foxly and crew have been on an epic journey of seemingly endless development on BP media, he claims to have solutions to problems, the source code has been visible for months, so why hasn’t core either adopted it and said lets get this into core (providing it only does this and you strip out the following) or publicly just say their code is crap avoid at all costs? Again is sharing media not one of the core components of a social network?
Even the forums are becoming fragmented, no method for searching for answers. Jeez move to a normal forum if you cant get it working, there is a requirement for eating your own dog food, but usually its a demonstration of how good something is not how broken it is!
Your users don’t want plugins.
We will tolerate them when it’s something a bit more specific but on the whole they are a risk we would rather not have to take… why… because unlike wordpress which is just a publishing platform, we have active users who become ‘dependent’ on the functionality and if it stops being maintained we are left in the lurch wondering how the hell we upgrade.
Fork it and move on
All being said this is pitched at @foxly – If BP Media can do what you say it can do and you feel that you can use your core to dramatically improve how buddypress works, then why not fork the project? Ditch the idea of BP Media being a plugin to a platform you have little say over and bundle it up as a complete Social platform in its own right? (I suppose the issue is it’s a plugin of a plugin right?)
A final Word
To be clear our team are massively appreciative of all the work Automatic, the BP Volunteers and the plugin developers have done. This is not an issue of individuals, but it does seem to be an issue of leadership, Buddypress will never flourish with a part-time leader, it needs full time management now as that would allow more core volunteers to be able to commit to the project, are Automatic planning on providing a full-time leader to Buddypress or does the community need to recruit one.
Buddypress is not a social network platform, i can safely say after 3 years and 2 years of having it in a production environment (there are times when it feels) that its just, Fancy Profiles for WordPress with a Ropey Private Messaging system.
Hopefully this isn’t dismissed as ‘just a rant’ or ‘yet another pointing out the obvious’ comment, as if it was obvious i’m sure this development blog would have topics relating to the above list aside from how cool CPT’s are ;o)
Peace!
Foxly 12:50 pm on March 9, 2012 Permalink
@Warzan
“I started reading this thinking your getting very cocky these days @foxly ;o) But to be fair, you do have a point.”
lol…maybe… but somebody has to get in there and shake things up …
“All being said this is pitched at @foxly – If BP Media can do what you say it can do and you feel that you can use your core to dramatically improve how buddypress works, then why not fork the project? Ditch the idea of BP Media being a plugin to a platform you have little say over and bundle it up as a complete Social platform in its own right? (I suppose the issue is it’s a plugin of a plugin right?)”
Simply put: there’s a lot more to running a successful open-source project than having fast code.
BuddyPress has an established brand, a large user community, and a dedicated hard-working dev team that’s stuck with it for years. Creating a fork of BuddyPress would spread that energy over two different projects, resulting in duplicated work and less progress being made overall.
Forking a project is a strategy of last-resort …only to be used when a project is abandoned by its developers or there are epic conflicts within the dev team.
Our strategy for getting our technology moved into the BP core is:
1) Release a working version of BP-Media to demonstrate our classes in action
2) Collect performance metrics on a wide range of live sites to tune our algorithms
3) Present our results to the BP team
4) Work with the BP team to refactor the BP core to use the new classes
^F^
Warzan 12:57 pm on March 9, 2012 Permalink
Sooooo… When’s it going to public beta?
Foxly 2:15 pm on March 9, 2012 Permalink
@Warzan
Public alpha this summer, beta in the fall, but other developers can already use the parts of the platform that we’ve finished. BP-Media is a huge piece of software.
^F^
Foxly 11:42 pm on March 9, 2012 Permalink
@Warzan
Also, what’s your site’s URL.
If you’ve got > 100K users, my team would be keenly interested in working with you to collect performance metrics and experiment with different methods to improve BP’s load times.
^F^
Stas Sușcov 12:18 pm on March 9, 2012 Permalink |
@Foxly
With all respect, your long post has almost no arguments regarding the 3 points you started with.
Ignoring the technical aspects you listed (which totally was not the case), I would really love to hear from you how exactly developers can not become attracted of a unified WP/BP API, and why CPT (even if are slower now in the BP context) cannot be improved, preserving the same old CPT API and making it just better/faster.
Judging from your comment, we should throw all away and start building it from scratch with the tools that are better (in your opinion).
Thanks.
Warzan 12:47 pm on March 9, 2012 Permalink
I can’t vouch for the correctnes of his post or the quality of his code. But his argument is that moving to CPT’s is not going to improve the situation.
1) It won’t be more simple
2) It won’t be any faster
3) It won’t attract any new developers to the platform.
He then goes on to link extensively to his source code to describe solutions they have built for Simplicity and Speed and its all designed to work with the current platform.
It doesn’t look to me like he suggested throwing anything away, but provided an alternative solution, rather than CPT’s.
As regards a unified API for WP/BP they are completely different applications built for different purposes, 1) Is built to manage the publication of content the other 2) Is built to handle a social community and how it interacts with that content.
Be careful that the lure of unification doesn’t cloud your judgement on the differences in purpose and requirements of these apps.
)
Now if someone would kindly give me the beef on whether @foxly code is up to snuff that would certainly shine a big light on the potential of this conversation ;o)
Foxly 2:04 pm on March 9, 2012 Permalink
@Stas Sușcov
“Ignoring the technical aspects you listed (which totally was not the case)”
Lets start with this one.
CPT’s are a “key”=>”value” storage system with serialized data and a dictionary table. CPT’s are great when you have a “one to many” relationship with objects that have custom data fields inside them …like a “blog post”.
Its possible to use CPT’s to store things like “items in an online store” and even to use WP’s taxonomy features to make them searchable “find me all the blue sweaters”, but that’s not what the system is designed for.
CPT’s can’t handle “many to many” relationships “find me all these users that are friends with all these other users”. They can’t handle data integrity rules “there can only be ONE user #17 in the database”, and they can’t handle transactions “Only add $50 to user #1′s account if we successfully removed $50 from user #2′s account”
In summary, blindly using CPT’s because “everybody is using CPT’s” is a recipe for disaster. BuddyPress has completely different data storage needs than a blog management system.
“I would really love to hear from you how exactly developers can not become attracted of a unified WP/BP API”
Because it won’t be unified.
The majority of the data BP has to manage can’t be handled using CPT’s. So the situation becomes:
1) “BuddyPress is currently handling 100% of their data using a complicated DB interface system”
2) “So let’s move 30% of that data to this other DB interface system”
3) “So now we have two different systems handling our data that developers need to learn”
4) “Profit?”
Now, somewhere in BuddyPress is going to be a piece of code that needs to do a JOIN on data that’s in the CPT system *and* in the old system. For example: “Find me all the activity stream posts from users that are friends with user #2″
So the BP team will need to write code that uses the old, complex system to access data that’s stored in the CPT system’s tables.
So at this point we have 3 systems in use:
1) The old system
2) The CPT system
3) The old-to-CPT bridge system
System #3 violates the “abstraction” principle in software design. If WordPress changes anything about the way their CPT’s work, it will break BuddyPress. So now, BP has to block users from upgrading their WP installs until they can verify that WP’s CPT implementation still works properly.
So you can see, how a simple idea like “I heard CPT’s are awesome. So I think BP should switch to CPT’s” turns into a big, messy problem.
“and why CPT (even if are slower now in the BP context) cannot be improved, preserving the same old CPT API and making it just better/faster.”
Because giving CPT’s the functions that BP needs would basically mean writing a SQL server, from scratch, inside of WordPress. And that’s just not something that’s necessary or practical to do.
CPT’s are designed to be a simple, convenient way to store arbitrary data objects in WordPress. They’re not designed to replace a SQL server.
“Judging from your comment, we should throw all away and start building it from scratch with the tools that are better (in your opinion).”
That is correct.
Do you still connect to the Internet on dial-up? Talk on a 1st generation iPhone? Or run your site on WordPress 2.1? Technology is temporary, and we can’t be afraid to move on.
We’ve thrown out three times as much code as we’ve ever released:
https://www.ohloh.net/p/buddypress-media/contributors/2079953877303961
You can’t get attached to code. It’s just code. The real value in BuddyPress is the community we’ve built and the knowledge we’ve gained about what makes a viable social networking platform.
^F^
Foxly 2:09 pm on March 9, 2012 Permalink
@Warzan
I’m not sure what you mean by “up to snuff”. It has full unit tests with 100% code coverage.
You can learn more about that here:
http://code.google.com/p/buddypress-media/wiki/DOCS_bpm_unit_test_top
^F^
Stas Sușcov 2:59 pm on March 9, 2012 Permalink
@foxly, If core team decides to move to CPT, I’ll jump in just to prove you that theoretical technical details you described are wrong (simply because that’s one of the ways you can see the problem).
I’ve built applications within key-value storage and I’m pretty sure all the overhead and changes are worth the trouble.
Once again, if it were to decide over: a hardcore social network system, or a niche easy to use and extend social network system. I would go for the second one. Always!
But thanks for the reply, interesting read.
Boone Gorges 4:22 pm on March 9, 2012 Permalink
@foxly – Thanks so much for jumping in here. I agree with most of what you’re saying. It is possible to make CPTs work for many-to-many relationships like those of BP content, but only through the use of wasteful and confusing taxonomy structures. (Check out this ticket for a long discussion of the issue in WP core https://core.trac.wordpress.org/ticket/14513).
I’ve been thinking a lot over the last few days (prompted by much of the discussion here) about the value of using as many of WP’s internal tools as possible. On the one hand, using those tools means that you get upstream improvements; on the other, it also means you have to deal with upstream changes/breakage, as Foxly points out. On the one hand, you have the potential to reduce the size of your codebase; on the other, it means you have to jump through a fair number of hoops to make the core schemas work for your data. On the one hand, by using WP’s internal tools, you create more potential overlap between those developers who work on WP and those who work on BP; on the other, it’s totally possible to build your application in layers, so that the top layers (like the content loops) look very much like WP, while the underlying structure is different.
I’m really looking forward to seeing what comes out of BP-Media’s alpha and beta, and to future collaborations between the BP-Media team and the BP core team.
Marcus 6:24 pm on March 9, 2012 Permalink
What I haven’t (don’t?) seen on here is which bits should be CPT’ed discussed…. the more I think about it, the less use I see in having CPTs in the case of BP.
CPTs are only useful in my opinion if it’ll make it easier for other plugins to hook in and do their thing. If it’s for caching, i don’t think that’s a worthwhile advantage on it’s own as you can make your own caching mechanism.
Which components would truly benefit from having a WP Post structure? Groups is really the only one I can think of.
Obviously a balance is needed, but I think consistency with WP (as much as possible at least) takes priority. Afterall, this is a WP plugin
Whilst Foxly’s project sounds impressive (look forward to seeing it), and says his will work at the million+ scale, but I doubt the majority of BP users are aiming for that scale of things, so it’s a lot of effort to benefit a small subset of power users.
Foxly 9:18 pm on March 9, 2012 Permalink
@Stas Sușcov
“If core team decides to move to CPT, I’ll jump in just to prove you that theoretical technical details you described are wrong (simply because that’s one of the ways you can see the problem).”
There’s no need to do that. The Finite Model Theorem (1974) basically states: “if you have a key-value store you can build any modern database structure”.
The real question is: why would you want to do that?
You’d effectively be competing against an open-source team that’s been at it for 17 years, that has 1.3 million lines of code behind them, and who have Facebook’s engineering staff contributing patches.
1) Custom post types takes (literally) thousands of man-years of work and reduces it to a key-value store with a dictionary.
2) Making CPT’s try to fill BuddyPress’ needs aims to add back much of that functionality by re-writing it in PHP …something that adds a 500% to1000% performance penalty and needlessly burns developer time battling very difficult problems that were solved by people with PhD’s forty years ago.
If BuddyPress tried to switch to a 100% CPT implementation, I wouldn’t be surprised to see 15,000 lines of additional code needed in the BP core. Although the code would use the familiar CPT functions, the way BP uses them would be anything but simple.
Put another way, there’s a BIG difference between knowing how to use foreach() and knowing how to use wp_db(), even though wp_db uses foreach() many, many times.
“Once again, if it were to decide over: a hardcore social network system, or a niche easy to use and extend social network system. I would go for the second one. Always!”
Consider this situation from a user’s point of view…
Should I go with BuddyPress, the social networking platform that guarantees my site can never grow into a profitable business, because it’s designed for small hobby sites?
Or should I go with Drupal, or Joomla, or Elgg, or Tiki, or BoonEx, or EngineY, or Telligent, or ImpressCMS, or Oxwall, or XOOPS, or SocialEngine, or dozens of other competing platforms.
If BuddyPress is going to win as a social networking platform, we need to prove we can scale to the million-user level. If we fail to do this, we’ll lose the people who can make the biggest contributions to the project before they even try it …they’ll pick another platform that can scale.
Foxly 11:10 pm on March 9, 2012 Permalink
@Marcus
“What I haven’t (don’t?) seen on here is which bits should be CPT’ed discussed…. the more I think about it, the less use I see in having CPTs in the case of BP.”
That’s because 95% of the people posting on this thread saying “use CPT’s in BP” have probably never even looked at BP’s source code.
“which bits should be CPT’ed”
To understand that, you need to take a high-level look at BuddyPress, its components, and the types of data that need to be stored.
Custom Post Types are good at storing well …posts… but looking at that in data modeling terms, this would mean:
a) an atomic data object
b) which optionally contains a closed set of sub-objects
c) that’s optionally part of a one-to-one, one-to-many, or many-to-one relationship with other data objects
d) which doesn’t have a dependency relationship with other data objects
e) and which is not updated frequently
If any of principle a) through d) are violated, we risk creating a deadlock
If principle e) is violated on a busy site, the system will probably start thrashing.
Although there are algorithmic solutions to these problems, they are so, so, far beyond what’s realistic to do in BuddyPress, they’re not even worth discussing.
====
BuddyPress has seven core components:
1) Core
2) Forums
3) Friends
4) Groups
5) Members
6) Messages
7) Xprofile
Core:
It might be possible to store various configuration settings in a CPT, but WP’s config store does a perfectly acceptable job.
BP-Media has a very advanced config class which includes an admin page engine, sanitizers, validators, and a 3rd-order keystore with caching and progressive load. We hope the BP team will eventually adopt our technology.
Forums:
BuddyPress’ forums are based on bbPress, which already uses CPT’s to store its posts. Our team has concerns about how this impacts the ability of bbPress to perform FULLTEXT searches of forum posts, and how this affects the performance of sites with large numbers of posts on their forums.
Friends:
Attempting to store a friend network graph using CPT’s would probably cripple any site with more than a few hundred users. This is because a network with N users could potentially have up to (N^2)/2 connections. Using CPT’s, this would require N rows in the datastore, and up to N^2 -1 rows in the dictionary. In other words, a site with 10,000 users could potentially require 100 million database entries.
The current solution used in BuddyPress offers acceptable performance for most users. Our team is putting a considerable amount of research into designing a solution that works at the 1 million user level (1 trillion connections) using a sparse-matrix caching algorithm
Members:
It would be possible to store members’ sign-up data in a CPT entry. However, every single field in a user’s profile that needs to be searchable would require its own row in the dictionary table.
So for example, if a user listed e-mail, firstName, lastName, country, state, city, age, programming languages, and “open source projects” …that would require 1 row in the “posts” table + 6 rows in the “dictionary” table + 1 additional row in the “dictionary” table for each “programming language” and each “open source project”.
The BP-Media team has created two different solutions for this problem. For complex user data (such as associative arrays) that doesn’t need to be indexed, our user data class offers a 3rd-order key-value store with a persistent paged cache and strong data validation.
For data that needs to be searchable, the BP team can use our BPM_db class to quickly create any table structure they want.
Messages:
It would be possible to store messages as CPT’s. However, handling messages sent to multiple recipients, flagging messages as spam, integration with groups, and system alerts would be immensely complicated.
We think time would be better spent hardening BP’s messaging code with transactions. This easy to do using BPM_db’s native transaction support.
Xprofile:
It would be possible to store members’ xprofile data as CPT’s, but it would create the same problems as trying to store their sign-up data using CPT’s.
This is a situation where BPM_db could offer a huge performance boost to BuddyPress. BPM_db can dynamically create tables in MySQL based on the fields the site admin enters in the profile setup. This allows fields to have their own column in the DB table, and as a result, be searchable.
====
Overall, we’re happy that the BP dev team has gotten to the point where they’ve realized they need a database abstraction layer.
Our team reached that point 12 months ago, but of course the stuff we’re working on is way more data-intensive. Since we created BPM_db we’ve been able to reduce the amount of code in our database classes by an average of 40%, massively cut our defect rate, and significantly improve our load times.
We hope that the BP team realizes they need far, far more than what’s offered by CPT’s, and we hope that they’ll either fork the BPM_db class and tune it to their needs, or team-up with us to make it even better.
^F^
Marcus 10:21 am on March 10, 2012 Permalink |
@Foxly: I think you have some good points here and I agree with a lot of what you say. Groups is still the only one to me that would make any sense, but then, is it even worth the hassle vs the benefits? Relationships to the groups/members should definitely be handled on a separate table.
Going through the list, my opinion
1) Core – makes no sense
2) Forums – n/a, the way it should be
3) Friends – asking for trouble as foxly correctly stated
4) Groups – see above
5) Members – no, but would personally love to see more integration with WP accounts
6) Messages – no
7) Xprofile – same as 5
Erlend 11:51 am on March 12, 2012 Permalink
If nothing else, Groups to me seems an ideal candidate for CPT based solely on the fact that I’ve practically seen no creative approach to groups styling whatsoever. Looking at all the crazy modifications people are doing using CPT, I’m hopeful the same would come true for Groups.
uloga 11:25 pm on March 7, 2012 Permalink |
I agree with Boone Gorges and John James Jacoby.however my opinion on the whole bp changes, it’s becoming messy’r and slower from version to version.
John James Jacoby 12:08 am on March 8, 2012 Permalink |
Thanks for your feedback, even though this isn’t exactly the place for it. If you have any data on what’s gotten slower and any code changes that improve performance, drop us a note on our development tracker.”/a>
Billy 11:26 am on March 16, 2012 Permalink |
The most important thing that needs to be addressed right now is user permissions and privacy for end users…. Very essential to be able to grow in numbers of people willing to join our platform…
David 11:10 pm on March 16, 2012 Permalink |
As someone who heavily uses CPTs, I can say that I love when code is reused, and 6 months ago, I moved all of our vBulletin forums over to bbPress simply because it was cleaner. I’d love to see Buddypress using CPTs, but…..
I know what a big problem that scalability thing is. We don’t even have a big site (~1,000 users). But when I moved all of that forum data, we ended up with a wp_posts table with 20,000+ rows. On my local test machine, this was REALLY laggy, especially when adding a few widgets like “Most Recent” and “Most Popular” topics to the page. Fortunately, we have a dedicated server, so the impact is hardly noticeable — maybe half a second. But this is with Buddypress and comments separated from wp_posts. What if everything was together? From a design perspective I like it, but I can’t disregard speed.
I look at this as more of an issue that WordPress needs to handle, not Buddypress. Maybe something with the CPTs or the way they are queried needs to be rethought. I wish I had the knowledge to know what to do. I know that hardware can solve or help most scalability issues, but still, at some point throwing everything into one huge table becomes less efficient, not more.
Dave 6:24 pm on March 22, 2012 Permalink |
It seems like if Buddypress content were moved over to custom post types, then Buddypress classifications (groups memberships, etc) could be made custom taxonomies. Makes for a clean, elegant system.
Erlend 8:57 pm on March 31, 2012 Permalink |
So it’s been almost a month since this massive discussion was started. I think a lot of *potential* good came out of it, so I just wanted to drop back in to inquire whether this was continued somewhere else, or even better if there’s been some code developments related to it already. There was talk of a merge of sorts.
Vittorio 10:50 am on May 21, 2012 Permalink |
Any update about this topic?
ericslangley 8:58 pm on May 31, 2012 Permalink |
As a non-programmer I found this topic fascinating and worth reading the whole thing. I came here (through Google search) because I am building a site that will have hundreds of unique fields for each member (an hopefully millions of members) with either Custom Post Type Custom Fields or BuddyPress extended profile fields. These fields need to be manipulated and presented to the users as part of the profile page and other pages on the site.
I am testing using Custom Fields using a plug in. Going this route would make them difficult to integrate with the BuddyPress profile page? Connecting them to the members profile?
I am also testing Buddypress Extended Fields but, as @foxly noted, I need image capability. Something lacking in Extended Profiles. Though there is this fairly recent hack; http://alextheafrican.wordpress.com/2012/03/10/how-to-add-an-image-field-to-buddypress-extended-profile-fields/
I’ll be curious to see how this progresses.
@elangley
~eric