• Posts: 157
  • Tag points: 716

Topic: Proposed New MapGen File Format

One of the main shortcomings of the current IC Map Generator tool is the fact that it only runs on Windows. At some point in the next few months, I'll hopefully get a chance to play around and write a similar tool in Java, which would run on any system that supports the Java VM (aka: most computers). The code for the tool would be released under the GPL and the project itself would be open to contributions from other players. I'm just doing a little brainstorming now about one aspect of the tool, namely the format for the map files.

The current IC MapGen stores its map data files as a binary blob. That's a space-efficient design (the ICPG map, for instance, is ~10 KB) but it is difficult/impossible to decipher unless you know what the bytes are supposed to represent. It's also not very extensible (if you look at the format, the last part is tacked on to allow for galaxies without an equal number of vertical and horizontal sectors), so the file format is very specific to the tool.

So, as an alternative, I propose an XML based format. It would be more verbose (and thus take up more space, although likely not exceeding 80-100 KB) but it would be more extensible and less domain-specific than a binary format. The format I've come up with so far is:

<map galaxy_id="1" galaxy_name="Milky Way">
    <description>Round 8</description>
    <sectors_x>2</sectors_x>
    <sectors_y>2</sectors_y>
    <systems_x>100</systems_x>
    <systems_y>100</systems_y>
    <system_url>http://www.imperialconflict.com/system.php?x=%d&y=%d</system_url>
    <family_url>http://www.imperialconflict.com/myFamily.php?fam=%d</family_url>
    <systems>
        <system x="1" y="2" homesystem="2048" />
        <system x="28" y="44" />
    </systems>
</map>

A key:
<map> is the container tag, it has attributes that give information about the galaxy that is represented. This would be useful if the tool wanted to change the display based on the galaxy (eg: the background of the display could be a transparent picture of the galaxy in question).
<description> is a description of what the current map is supposed to represent. In the current IC MapGen, it's used to allow users to identify the map from within the program.
<sectors_x> and <sectors_y> simply give the number of sectors in the galaxy
<systems_x> and <systems_x> simply give the x/y boundaries for coordinates.
<system_url> and <family_url> are URLs used to load external pages, either system pages or family pages.
<systems> is a container for the individual <system> tags.
<system> tags represent systems on the map. They have x and y attributes that represent the co-ordinates and an optional homesystem attribute which specifies which family's homesystem it is.

Now, I'm posting all of this because I would like feedback:
Does XML sound like a good choice? Does this file format look OK? Am I forgetting some data that the tool would need? Does some of this not make sense? wink

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points

  • Posts: 5,988
  • Tag points: 241

Re: Proposed New MapGen File Format

I'm not experienced in this matter much - so somebody will have to advise, but to get you started!!

> Does XML sound like a good choice?
Yes

> Does this file format look OK?
Yes

> Am I forgetting some data that the tool would need?
Any other data that would require you to implement some of these features discussed in making a new MapGen?
http://www.imperialconflict.com/forum/viewtopic.php?id=47821

> Does some of this not make sense? wink
Yes! tongue

0 Dislike 0 Post Points

  • Posts: 157
  • Tag points: 716

Re: Proposed New MapGen File Format

lol tongue

Well, that's what I get for not reading the forums thoroughly: I had no idea other people were working on the same thing! smile

We'll see if I even get around to writing the program. I'm just brainstorming at this point.

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points

  • Posts: 5,988
  • Tag points: 241

Re: Proposed New MapGen File Format

> Smartys wrote:

> We'll see if I even get around to writing the program. I'm just brainstorming at this point.



...err....have you been hanging around Stefan a bit much lately? wink

0 Dislike 0 Post Points

  • Posts: 157
  • Tag points: 716

Re: Proposed New MapGen File Format

If I had been, I would write a prototype with most of the features in place and then get bored with it and scrap it. And it wouldn't be open source. tongue

No, I just have other projects that take a higher priority. wink

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points

  • Ps
  • Player
  • Offline
  • Posts: 3,725
  • Tag points: 10

Re: Proposed New MapGen File Format

I'd propose using height and width, as it's more intuitive then systems_x/systems_y

Rehabilitated IC developer

0 Dislike 0 Post Points

  • Posts: 157
  • Tag points: 716

Re: Proposed New MapGen File Format

Hmm, I don't know. Height/width seems less intuitive to me.

A. It's not like people are generating this by hand, they'll be using a script. The tag names are less important in that respect.
B. What exactly is the "height" of a galaxy? What units is this height in? Height/width are very generic, non-specific terms.

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points

  • Posts: 0

Re: Proposed New MapGen File Format

i think that it should become 3-d, and cords planed by 6 digit combos, but plans can be changed and stoped witout total loss of spent... like if you recall an attack that has gotten 1/4th to the place you lose one half what was spent, if you build droids and it takes two ticks you loses half.
and buracracy... they'er isint one, let us fight... and have means to do it.....

0 Dislike 0 Post Points

  • Posts: 5,988
  • Tag points: 241

Re: Proposed New MapGen File Format

lol'd

0 Dislike 0 Post Points

  • Posts: 7,941
  • Tag points: 14

Re: Proposed New MapGen File Format

I shall post on this later. This is an interesting topic, and I think I can contribute.

Everything bad in the economy is now Obama's fault. Every job lost, all the debt, all the lost retirement funds. All Obama. Are you happy now? We all get to blame Obama!
Kemp currently not being responded to until he makes CONCISE posts.
Avogardo and Noir ignored by me for life so people know why I do not respond to them. (Informational)

0 Dislike 0 Post Points

  • Posts: 2,453
  • Tag points: 4142

Re: Proposed New MapGen File Format

yes i agree, you should post later on this

<@Nolio> Ilu was the man back in the day,he even made monkeywrench and arganon look good for half a round =p
<@iluvatar> it is my grandest achievement
<@Nolio> *half a round  =p
<@iluvatar> still
* Final_Doom is now known as Thanks_Iluvatar

0 Dislike 0 Post Points

  • Posts: 7,941
  • Tag points: 14

Re: Proposed New MapGen File Format

Ok so now to make the post.

The galaxy is not made up of every other planet, nor in a discernable pattern.

Yeah we could store coords in some binary format, but that makes it difficult to 'reverse engineer'.

Instead there is an easy way. Even with 1,1 to 100,100 there is only 10,000 actual coords. Sadly due to base 8 this stores as a much higher sized text file, with it being 10kb. But if this is acceptable we can easily store coords for a .dat galaxy file as a 0 for empty space, 1 for a planet, a 2 for next row no planet, a 3 for next row planet, a 4 for home system, 5 for home system next row. Or letters can be used. Each family could have an allowed number space inside ' ' brackets, for instance '4422' Even with 100 families this represents a mere 600 bytes more in file size.


Additionally if space is an issue, we could even go much more extreme, and count til star system. 0 = next to current, 1 is a 1 tick gap, 2 is a 2 tick gap, up to 8, 9 standing for no stars within 8 ticks, start again. This would still be easily understood, and a letter code could follow the star to stand for either home or general star system.


Now the mapgen file you make when doing families is a different issue. In truth we need only know which of the many different plists is being used, and in which system. Additionally a planet number must be stored. If there is 1000 systems we already know their proper order via the map galaxy file, so we need only go through a system listing the different people in each one.

If we use A through Z, upper and lower cases, we can store 52 players. This may not be enough for most people however, so using symbols excepting two could add an additional 30 players (Standard keyboard characters).

The way this could work is if the player Y and $ share the third system, and no other players exist, then we use a symbol for 'empty system', another 'empty system', then the Y130104 (13 standing for planet number in there, followed by 01 for planet 1, and 04 for planet 4.) with a $050607 (Standing for the other player owning planets 5, 6 and 7.). No additional symbols are needed so the symbol for the empty next system would occur, until the entire map is taken up.


Further size reductions are possible ofc, this is beyond basic, but this leaves it so a person in theory could write their own file. Additionally at the front of the file would be a section for the 82 different player files to be marked as ally, enemy, family, as well as a fixed size length name description (as is currently used for name of the player and family, or just family in yama usage) allowance. If fixed at 32 characters for friend/foe/family and for description we can suit or needs with a maximum of 2624 bytes here.

Actual file size of course would rely upon the number of stars and number of entries incorporated.

I believe that a this would require a built in sort on the coords and would list them in proper order when printed for the player under the circumstances due to how the files would be stored.


Under the current system I know that it tracks a variable sized amount of information in a saved file, parses the information seeking the required 3 to 5 digits information per star system per running of the application, and also saves about the same title length.

In comparison I believe the current is more capable with smaller planet lists, but as the number and size of players and their planets increase my system would see a marked decrease in file size versus current save files. Additionally without excess information stored (prints to the program efforts can be parsed immeadiately to remove excess information) this could result in further savings.


The idea I believe, as you stated above, is to make it simple so others can do whats needed if there is an issue with the programmer disappearing for a while. I believe this would make that possible.

Everything bad in the economy is now Obama's fault. Every job lost, all the debt, all the lost retirement funds. All Obama. Are you happy now? We all get to blame Obama!
Kemp currently not being responded to until he makes CONCISE posts.
Avogardo and Noir ignored by me for life so people know why I do not respond to them. (Informational)

0 Dislike 0 Post Points

  • Skoe
  • Skoe
  • Player
  • Offline
  • Posts: 15

Re: Proposed New MapGen File Format

> Does XML sound like a good choice?

For me? No. For you? Probably not.

For all of us? Yeah, i'd say so.

> Does this file format look OK?

It looks fine.

> Am I forgetting some data that the tool would need?

Probably. But that wouldn't matter with XML... wink. Just call it the first standard revision, more
things can be "standardized" later.

> Does some of this not make sense?

Flints post...ergh.

Morbo: Morbo can't understand his teleprompter. He forgot how you say that letter that looks like a man with a hat.
Linda: It's a 't'. It goes "tuh".
Morbo: Hello, little man. I will destroy you!!
http://www.youtube.com/watch?v=cpP7b2lUxVE

0 Dislike 0 Post Points

  • Posts: 1,355
  • Tag points: 54

Re: Proposed New MapGen File Format

Flint - did you read smartys post.

What you're suggesting is a binary file very similar to waht we have now, Smartys has already said that he thinks XML is better because although more verbose it is extensible and easier to decipher.

Your suggestion is for an even denser and less decipherable file yikes

0 Dislike 0 Post Points

  • Skoe
  • Skoe
  • Player
  • Offline
  • Posts: 15

Re: Proposed New MapGen File Format

> The current IC MapGen stores its map data files as a binary blob. That's a space-efficient
> design (the ICPG map, for instance, is ~10 KB) but it is difficult/impossible to decipher unless
> you know what the bytes are supposed to represent. It's also not very extensible (if you look
> at the format, the last part is tacked on to allow for galaxies without an equal number of vertical
> and horizontal sectors), so the file format is very specific to the tool.

Its optimized for speed too: You read a piece of data, thats it. You already know what that data
does.

With XML you have to read the entire file in, and as you parse you don't actually know what something
does: E.g. '<' means something different to '\<'. This is TRIVIAL because your underlying API (whichever
you use) takes care of the parsing.

> So, as an alternative, I propose an XML based format. It would be more verbose (and thus take up
> more space, although likely not exceeding 80-100 KB) but it would be more extensible and less
> domain-specific than a binary format. The format I've come up with so far is:

The only numbers you really need to worry about are the systems, which would be the killer. But
i don't think people will care even if the mapfiles are 4-5x their current size, so theres really nothing
to worry about.

Morbo: Morbo can't understand his teleprompter. He forgot how you say that letter that looks like a man with a hat.
Linda: It's a 't'. It goes "tuh".
Morbo: Hello, little man. I will destroy you!!
http://www.youtube.com/watch?v=cpP7b2lUxVE

0 Dislike 0 Post Points

  • Skoe
  • Skoe
  • Player
  • Offline
  • Posts: 15

Re: Proposed New MapGen File Format

> Your suggestion is for an even denser and less decipherable file

His suggestion is garbled. It doesn't make any sense...

Morbo: Morbo can't understand his teleprompter. He forgot how you say that letter that looks like a man with a hat.
Linda: It's a 't'. It goes "tuh".
Morbo: Hello, little man. I will destroy you!!
http://www.youtube.com/watch?v=cpP7b2lUxVE

0 Dislike 0 Post Points

  • Posts: 157
  • Tag points: 716

Re: Proposed New MapGen File Format

Flint: As others have said, you just completely ignored my post. You're proposing a binary file format very similar to what already exists.

"Instead there is an easy way. Even with 1,1 to 100,100 there is only 10,000 actual coords. Sadly due to base 8 this stores as a much higher sized text file, with it being 10kb. But if this is acceptable we can easily store coords for a .dat galaxy file as a 0 for empty space, 1 for a planet"
This is exactly how the current format works (with extra data before/after for family numbers, homesystems, and general galaxy data).

skoe: Thanks for the input

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points

  • Skoe
  • Skoe
  • Player
  • Offline
  • Posts: 15

Re: Proposed New MapGen File Format

> Sadly due to base 8 this stores as a much higher sized text file

base 8 has to do with this what now how??!??

Morbo: Morbo can't understand his teleprompter. He forgot how you say that letter that looks like a man with a hat.
Linda: It's a 't'. It goes "tuh".
Morbo: Hello, little man. I will destroy you!!
http://www.youtube.com/watch?v=cpP7b2lUxVE

0 Dislike 0 Post Points

19 Reply by Ps

  • Ps
  • Player
  • Offline
  • Posts: 3,725
  • Tag points: 10

Re: Proposed New MapGen File Format

Be nice to Flint tongue We all say retarded things now and again.

The increased file size is not too big of an issue. If anyone was that worried, xml files compress very very well and could be distributed zipped.

Although future mapgen replacements should have the ability to make a map file on there own, so whether or not the mapfile format is tool specific or not will be irrelevant.

Rehabilitated IC developer

0 Dislike 0 Post Points

  • Posts: 157
  • Tag points: 716

Re: Proposed New MapGen File Format

"The increased file size is not too big of an issue. If anyone was that worried, xml files compress very very well and could be distributed zipped."
A very good point! I'm going to go test it now and see how much space gzipping saves.

"Although future mapgen replacements should have the ability to make a map file on there own, so whether or not the mapfile format is tool specific or not will be irrelevant."
I don't know that I agree with that.
A map file can be created in a variety of ways. It can be constructed using an image of the map, the actual HTML, a coordinate list, etc. Ideally, it would be constructed by the game itself. However, it should only need to be done once. A map file generator seems like it would be very good as a standalone tool for people who want to design their own galaxies, but I think it's somewhat outside of the scope of the general tool.

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points

21 Reply by Smartys (edited by Smartys 14-May-2009 03:26:29)

  • Posts: 157
  • Tag points: 716

Re: Proposed New MapGen File Format

I wrote a test script that generates a "worst case" XML file (one where all possible systems exist and every system is a homesystem owned by a unique family). For a 200x200 map, that meant 40,000 systems (no galaxy is ever this full of systems). This led to an uncompressed file that was ~1.75 MB. Gizpping took the file size down to ~140 KB. Other tests with less densely populated files led to similar gains. Hurrah!

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points

  • Skoe
  • Skoe
  • Player
  • Offline
  • Posts: 15

Re: Proposed New MapGen File Format

> Gizpping took the file size down to ~140 KB. Other tests with
> less densely populated files led to similar gains.

Are you proposing that individual applications understand how to unzip
this archive, or are you saying that the distribution of these map files
can be done zipped -- and then the user can unzip it themselves?

You will more then likely get better results with the 7-zip too.

Morbo: Morbo can't understand his teleprompter. He forgot how you say that letter that looks like a man with a hat.
Linda: It's a 't'. It goes "tuh".
Morbo: Hello, little man. I will destroy you!!
http://www.youtube.com/watch?v=cpP7b2lUxVE

0 Dislike 0 Post Points

  • Posts: 157
  • Tag points: 716

Re: Proposed New MapGen File Format

I don't think it's unreasonable for applications to be able to handle gzipped data files. If a particular application does not, the user can always decompress the file themselves.

I'm not familiar with 7-zip: is it as widely available and as easy to decompress as plain gzip?

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points

  • Skoe
  • Skoe
  • Player
  • Offline
  • Posts: 15

Re: Proposed New MapGen File Format

It wouldn't be anywhere near as easy i guess tongue. Compression wouldn't be anywhere
near as important as ease-of-use for the end-user, so IMHO distributing XML files without
any compression would be best. This would be up to the distributor though and wouldn't
really need to be taken into consideration right now...


I also just realized that the above XML fails to identify what 'game' the map is for.

Morbo: Morbo can't understand his teleprompter. He forgot how you say that letter that looks like a man with a hat.
Linda: It's a 't'. It goes "tuh".
Morbo: Hello, little man. I will destroy you!!
http://www.youtube.com/watch?v=cpP7b2lUxVE

0 Dislike 0 Post Points

25 Reply by Smartys (edited by Smartys 14-May-2009 05:10:43)

  • Posts: 157
  • Tag points: 716

Re: Proposed New MapGen File Format

Mmm, I don't see how the compression is very complex to deal with from the programmer's perspective though. There are plenty of libraries to handle it, many languages come with classes or libraries built in (Java has java.util.zip.GZIPInputStream, C# has a GZipStream Class in System.IO.Compression, PHP has zlib, etc). From the end-user's perspective, the process should be transparent. But you're right, it's not something that's very important at this point.

And what do you mean when you say it doesn't identify what game it's for? The XML doesn't have some place that explicitly says IC, but does it really have to? The URLs in the XML allow the program to work with multiple different URL schemes (or games). It might be useful as meta-data, but then who determines what is and isn't a valid value? Me? wink

CLEAR YOUR CACHE!
http://www.microsoft.com/windows/ie/usi … cache.mspx
Do not read this fortune under penalty of law.
Violators will be prosecuted.
(Penal Code sec. 2.3.2 (II.a.))

0 Dislike 0 Post Points