Wad compatibility option database - ideas

[Graf Zahl]

Xaser said:

To me, this is the sort of thing that makes more sense to have on a single, centralized server. But then it boils down to who's gonna run it.

... which sounds nice but has its own pitfalls. Imagine some jerk messing it up but since it gets used directly off the server everybody playing will be affected. I personally vote for strict offline use which involves far less risk of something getting broken.

[DaniJ]

That would only happen if we stupidly tried to implement this without using collaborative community authoring rules (like a wiki) on the central database.

Storing compatibility.txt in a versioning file system would technically accomplish that however you then push all the logistical management issues onto the community.

[Xaser]

A wiki-like approach wouldn't be what I'd have in mind for this, unless each "post" is approved by a moderator (e.g. port author) of some sort. That'd prevent nonsense entries and someone needs to 'control' the system anyway, else it'd be all sorts of chaotic.

[DaniJ]

That is not what I meant. I meant the solution should use collaborative authoring rules, like a wiki does, not that it should be implemented as a wiki.

[Graf Zahl]

Of course, but that still means that without some moderation it's not going to work, especially having ports download unmoderated information. The rist of vandalism is too high.

So the end result remains: It makes no sense to automate the entire thing. Whatever the ports get from the database must be approved by someone.

[DaniJ]

Sorry but I don't buy the vandalism argument. What is there to gain from changing a few compatibility bits? Its not like users would be able to inject adverts for Viagra or hyperlinks to warez sites. Why can't this be left to the community to police? It seems to work well enough for wikis.

Also how do you plan to allow users to submit updates and then incorporate them into the "database"? Furthermore, how do you plan to allow users to search within said database?

[Graf Zahl]

DaniJ said:

Sorry but I don't buy the vandalism argument. What is there to gain from changing a few compatibility bits? Its not like users would be able to inject adverts for Viagra or hyperlinks to warez sites. Why can't this be left to the community to police? It seems to work well enough for wikis.

As long as the user is in full control of getting the data that would be fine. However, if the ports are supposed to download this off the server directly that's just not good enough for me and I would never advocate using such a service implicitly.


As for the vandalism argument, what's the point in general about vandalism? There is none and yet it exists. And it will also exist here because somewhere some jerk will just sit and wait to spread chaos. These types don't act rationally so your rational arguments don't apply.

[DaniJ]

What I'm saying is that disregarding wanton acts of mindless vandalism - there is no incentive to vandalise. Unlike wikis where "vandalism" can achieve something (e.g., piping more users to your drug store) there is no reason to vandalise the database. If one examines the amount of vandalism inflicted on popular wikis it can be observed that the vast majority of it is from users who have something to gain. In this instance there is nothing to gain so therefore the risk of vandalism is significantly reduced.

So long as there is a mechanism in place to counteract mindless vandalism, I see no reason not to automate this as far as possible.

If you have issue with downloading the latest copy of the database from within your port - you are quite free to instead implement the update manually in your release system.

[Gez]

Wiki vandalism, like forum vandalism, is mostly created by automated spambots, which are designed to register, log in, and edit automatically. That's why popular software (e.g., MediaWiki, phpBB) are targeted a lot more than custom-built software. So a situation like that of the Eternity or Vavoom wikis, completely overrun by spambots, is presumably not something to be afraid there unless we do something dumb like turn compatibility.txt into a MediaWiki page.

[DaniJ]

Exactly. People don't write spambots because they simply like defacing the internet with garbage. There is a purpose behind that spam.

Potential-for-vandalism is no reason to fundamentally compromise the design of the central database. Unless some individual volunteers to personally populate the database and maintain it - we absolutely must have a usable mechanism for the community to do so.

[kb1]

Xaser said:

...why not use a proper DBMS?...a single SELECT statement...

That single SELECT statement has to parse the whole file too, and/or build an index which has to parse the whole file, at least once. And, that database will be bigger than the text file, guaranteed. I wouldn't want to add a db engine to a port, either, though, in a launcher it could be useful.

Xaser said:

To me, this is the sort of thing that makes more sense to have on a single, centralized server. But then it boils down to who's gonna run it.

Doesn't work for me - I have no internet at home. Which really sucks because I have a $60 Rage DVD set sitting in a box doing nothing...

Xaser said:

An extended version of ZDoom's compatibility.txt (e.g. one large text file) seems like it might get large after a while, and having to either maintain a copy on the user's hard drive or read the whole file from the server each time sounds like it'd be a pain.

Ok, here's my smart-ass comment :) Take that pain, and subtract all the potential pain caused from playing 100's of wads with incorrect settings...bla bla bla... For the file to get unreasonably big, many, many people would have to submit map corrections, and, by necessity, there would have to be that many maps with problems to correct.

People said:

...vandalism...

Yes, there's a concern with an automated type of update system. I am of the position that, we need to learn to crawl before we walk, and walk before we run. If a forum were to exist, where people would submit requested changes, I would maintain the text file manually. It's not difficult, because it's a text file. But, once again, getting a sizeable chunk of data is step #1.

[Graf Zahl]

DaniJ said:

Exactly. People don't write spambots because they simply like defacing the internet with garbage. There is a purpose behind that spam.

Potential-for-vandalism is no reason to fundamentally compromise the design of the central database. Unless some individual volunteers to personally populate the database and maintain it - we absolutely must have a usable mechanism for the community to do so.

You are naive.
Sorry to say that but why do some people vandalize wiki pages, why do others destroy traffic signs, smash windows or other purely destructive things?

The same would happen here. Just have a look at the multiplayer community. There's enough braindead morons there that do damage for the sake of doing damage. And if they find a system that's this easy to vandalize they will do - guaranteed. It only takes one vandal in a group of 1000 to destroy the entire thing.

If such a system gets set up without any safeguards I wouldn't use it and I wouldn't add support to my port for it because I don't trust it.

[Blzut3]

Gez said:

That's why popular software (e.g., MediaWiki, phpBB) are targeted a lot more than custom-built software.

Would be worth noting that spam bots have, for awhile now, been able to parse HTML forms. My website, which uses a custom CMS, has been subjected to spam even after obfuscating the field names.

That said, for Skulltag I've added the bug tracker to the robots.txt file and it has been relatively free (one known attack) from vandalism thus far (unless it's being moderated so well I never notice it).

[DaniJ]

Where did I say "Gee its Ok, we can trust that nobody will try to abuse the system", or how about "We don't need to implement safeguards for this because the data isn't worth defacing"?

Are my posts really that hard to follow? Am I talking another language? I'm really beginning to think so.

kb1 said:

That single SELECT statement has to parse the whole file too, and/or build an index which has to parse the whole file, at least once. And, that database will be bigger than the text file, guaranteed. I wouldn't want to add a db engine to a port, either, though, in a launcher it could be useful.

Please do not take offence but have you even used a relational database? "That SELECT statement has to parse the whole file too" eh? If we implemented this in the most cack-handed manner known to man - yeah I guess it would. However nobody in their right mind would implement the database component as a "map" of the flat file :p

[kb1]

DaniJ said:

Please do not take offence but have you even used a relational database? "That SELECT statement has to parse the whole file too" eh? If we implemented this in the most cack-handed manner known to man - yeah I guess it would. However nobody in their right mind would implement the database component as a "map" of the flat file :p

Uh, yeah, every day. I've also written a couple of database engines. No offense taken. But, yeah, the data must be entered into the database, so, yeah, it has to be read from the source, and it's expected that a SQL SELECT can find any one of the records based on it's criteria. Sure, once it's in the database, the database uses it's native methods of storage and retrieval, whcih you would hope used a hash lookup, but it's *responsible* for parsing every entry, even if it uses an optimized way to retrieve the data (indexes, which get built by parsing all the data). And, when you mount the database, the data files do get loaded into memory (possibly on demand). A db engine is traditionally optimized for retrieval, whcih means it is built to care a bit less about memory usage, CPU usage, etc. Of course, those things are also considered.

Not so with a sequential file read. Parse each line, stop when you hit the proper record. We could even sort the hashes to allow a binary search, which is implemented rather simply.

The point is, locally, do I want that database engine running on my gaming machine, just to facilitate a once-per-map lookup? Not particularly. It's unnecessary. Reading the file will do just fine. The OS will probably cache it anyway. When you load a mapset, the next map is probably the next set of lines. Once again, I can parse 1,000,000 lines/second without trying too hard on my machine, cached or not.

Still missing the point: Come back to me when there's 10,000 maps listed in our dataset - then we can talk about advanced storage/retrieval etc. You don't start with the Cadillac, you start with the most basic setup that sufficiently gets the job done, that's just using common sense.

[DaniJ]

Why would you need to implement a database engine on every machine that is to use the data? I thought we had already established that this would be unnecessary. If the central database represents this data using an actual database this does not immediately mean that it would be served to clients as a bunch of SQL statements (for you to rebuild locally). It could instead be served in exactly the same format as used in ZDoom's compatibility.txt.

If you want to parse this database as a whole and then do your own transforms into an efficient scheme for runtime use, to enable you to search within and manipulate it, then fine - knock yourself out. However if you stop to consider the whole problem rather than just your own needs you would realize that the efficiency transform can be offloaded to the central database.

I do not accept your "but we don't have enough data yet to worry about it" explanation as to why we shouldn't even attempt to take the design job seriously and implement a proper solution.

[kb1]

DaniJ said:

Why would you need to implement a database engine on every machine that is to use the data? I thought we had already established that this would be unnecessary. If the central database represents this data using an actual database this does not immediately mean that it would be served to clients as a bunch of SQL statements (for you to rebuild locally). It could instead be served in exactly the same format as used in ZDoom's compatibility.txt.

I assumed that Xaser was describing a local database. Of course a server implementation can be fat, and can spit out a number of formats to clients as needs demand. And, that *can* be created incrementally and empirically, as demand increases. I'm still waiting to see if we've scared everyone off with incessant bitching, to the point that no one will participate in adding data. That's what I need as proof - to see that people will find value in the data, and value in creating the data, before I build some monster server app/database. Starting small *is* a serious implementation of proof-of-concept, and of a trial run. As I stated above, I'll volunteer to maintain the data if I know what data there is to enter. If and when it becomes popular/widely supported/etc, we will know *exactly* what is required. Otherwise, yeah, build a big web site, super-slick database, and maybe find out that only 5% of it gets used, or that something was missed.

Take it for what it is - right now, it's a handful of map definitions that can be read by ZDoom and friends. No other source ports can read it. We've spent a boatload of time discussing formats and files and databases, very academic, but not very helpful. I would hate to send you and others off on some big coding project, for something that flops. Right now, being able to get feedback from the community, and implementing a "quick-and-dirty" update process *is* the proper solution. If it flops, no major work was lost. If it's successful, we can consider building the best solution yet.

[DaniJ]

Ok, I can accept that you would rather implement a prototype first. So in the interest of moving this forward - please lay out how you would have us progress with the implementation, on both client side (say a source port) and the central database, using ZDoom's compatibility.txt as the chosen representation. Tell me how you intend to implement the update mechanism, the user-facing interface, the client side management systems etc...

[Xaser]

@DaniJ: Earlier, I meant "wiki-like" in the sense of "anyone can freely contribute directly", not as in "use an actual wiki software" (hence the quotes around "post"). The way I'd envision the general-use front-end is a simple submission form, of which the submissions are later approved/rejected by a moderator into the database itself.

Graf Zahl said:

So the end result remains: It makes no sense to automate the entire thing. Whatever the ports get from the database must be approved by someone.

That sounds almost perfect, but backwards: Whatever goes into the database must be approved by someone. Makes more sense to have the input moderated, not the output.

kb1 said:

I assumed that Xaser was describing a local database.

Wait, how'd you miss the part where I was complaining about doing things locally? You replied to it directly. :P

No, if it's done locally, a DBMS would be sheer overkill. The 'perfect' compromise here would be to have a DBMS on a centralized server that the participating ports can contact, which can also allow users to extract snapshots that can be referenced offline. But then that requires the port authors to effectively implement two ways of loading things. Only a win-win if some are willing to do some extra work. :(

[DaniJ]

Xaser said:

@DaniJ: Earlier, I meant "wiki-like" in the sense of "anyone can freely contribute directly", not as in "use an actual wiki software" (hence the quotes around "post").

I see, my bad.

The way I'd envision the general-use front-end is a simple submission form, of which the submissions are later approved/rejected by a moderator into the database itself.

Yeah, that is the way I expect this part of the problem to be addressed.

That sounds almost perfect, but backwards: Whatever goes into the database must be approved by someone. Makes more sense to have the input moderated, not the output.

Indeed. Why force all ports to repeatedly rationalize the whole database at snapshot-time, when you can instead do so on a component-wise basis at update-time.

The 'perfect' compromise here would be to have a DBMS on a centralized server that the participating ports can contact, which can also allow users to extract snapshots that can be referenced offline. But then that requires the port authors to effectively implement two ways of loading things. Only a win-win if some are willing to do some extra work. :(

Got it in one Xaser - that is how I had envisioned this working from the outset.

[kb1]

Xaser said:

Wait, how'd you miss the part where I was complaining about doing things locally? You replied to it directly. :P

I have no idea...Disregard please :)

DaniJ said:

(agreeing with Xaser)

Also agree :)

DaniJ said:

...So in the interest of moving this forward - please lay out how you would have us progress with the implementation, on both client side (say a source port) and the central database, using ZDoom's compatibility.txt as the chosen representation. Tell me how you intend to implement the update mechanism, the user-facing interface, the client side management systems etc...

Oh boy, I'm in the hot seat. Very carefully?

Seriously, internet-facing interfaces are not my speciality, although I've built a couple. Earlier I mentioned that I would need some time to be able to move forward - that hasn't changed much, unfortunately. But, if I were following the "quick-and-dirty" scheme, I might go so far as to suggest, for version 1.0, that the compat.dat (8.3 names, please) file be updated by hand, by me, or a select few others, based on posts by trusted members of a forum thread. Tedious? Maybe, if we see some real participation - which would be a good thing. A link would be provided for the latest version.

Yep, that's about as "bare-bones" as you can get, but, assuming that a forum exists that wouldn't mind a sticky thread in support of this project, it would really be easy to implement - at first.

But, I don't claim that this is a great idea, or that I have the knowledge or time or ability to produce something better. In fact, I can think of ways to make this thing really robust, but, again, I think we should start small, for everyone's benefit.

My RL time constraints make it difficult for me to spend too much effort designing a robust solution, so I absolutely welcome ideas. However, I could spent some time here and there doing some "grunt work", updating a file/database.

So, in summary, to "get started", I think we need a sticky thread on a forum for submissions. The thread could be searched for immediate map playing needs, until we get some source port support. It looks like ZDoom will be the easiest and possibly first to adopt the improved compat.dat data. But, we'll need data to convince all devs to include support.

[DaniJ]

When you said prototype I was kind of expecting, well you know, a prototype. It sounds to me like you could just as easily implement that using a combination of MediaFire and a Gmail account.

I fail to see how this achieves anything. We already know the data is useful and that it benefits users. What do we stand to learn about the problem by implementing a solution like that?

[Xaser]

@DaniJ: Ah, good to hear we're both on the same page. :)

In response to some recent points:
Having someone upload a compat.dat would sorta circumvent the idea of having a DBMS on the server. Having a compat.dat generated by the server would be nice, though. ;)

My vision of the components of the full setup (sans some implementation details, e.g. how to get the hashes from the user):

• Database backend. This could easily be a MySQL/PostgreSQL install whose purpose is to actually store the compat data. We'll figure out the actual schema once we go that route. ;P
  
• Public submission page. This would be a simple HTML+whatever front end where people submit a list of compat options for a particular wad/map. Depending on spambot fears, we could take it a step further by requiring users to register a tiny account first, which has its own set of pros and cons.
  
• Moderation page. This is where the 'moderators' of the system can log in and see a list of pending submissions. They can approve submissions here (or, of course, reject them), after which the compat options go into the database system itself. A really robust mod page would also allow easy viewing and editing of what's in the database itself, though that itself could also be achieved by just allowing mods access to the database. :P
  
• Query system. This would be a tiny online app that would accept a hash value from a source port and serve back the list of compat options, for ports/authors that opt to get the options directly online.
  
• Compat.dat generation tool. Though it's less of a "tool" than a "thing users can use from the public submission page", this would generate an offline-use compat.dat file from the current database. We could forgo this option and instead have someone in charge of creating public "official" releases of compat.dat, if that options' more desirable.
  

I'm not exactly a guru (though I have set up a system like this before at work), but it doesn't seem like it'd be difficult to achieve once some server space is secured. As mentioned before, most of the work, I'd imagine, would come from the port authors' side. Just a guesstimate, though, coming from me. Thoughts on this?

For hosting, I'd probably contact Manc, though either way I'm positive there's someone here who wouldn't mind donating some space (and perhaps assisting with setting up the backend, since I for one am rubbish at that).

A sticky forum thread is cool for discussion, though it'd primarily have a link to the submission page and instructions to use that.

Note that I've got more ideas for the finer details. I just don't want to post a wall of text to soon. :P

[kb1]

I would - I would go bare-bones on it, just to see if people would participate enough to justify the big production. But, I could be wrong. I was thinking that people posting would not need to know the exact syntax and mechanics - I was considering converting their raw posts into properly-formatted lines - you know, bare-bones approach. I'm not stuck on that idea though, I simply don't want DaniJ or Xaser (or me for that matter) to work their ass off on a big system that gets 10 submissions - that was my only concern with building the big server too early. I didn't mean "proof that it's helpful", I meant "proof that people will help" :)

[DaniJ]

I appreciate the concern but by not implementing any user-facing interface at all - you actually undermine the collection of the metrics you are hoping for. This is one part of the prototype really has to be functionally complete otherwise you sabotage its very purpose.

[Gez]

Typical chicken & egg case.