XF86Config file First a WARNING: defining a bad MemBase may cause serious injury or death (to your operating system, of course). Especially defining the MemBase to be inside the range of system memory is a ticket to hell.
Rule #1: first, let the server find a memory base by itself, without specifying it. Make sure you "sync" all files to disk and close all critical applications. Make sure nothing bad will happen to your filesystems if you have to jump for the power switch soon.
The most critical cards are the ET4000W32p rev a and rev b on VESA local bus (VLB). The server will autodetect a linear base address that doesn't work on all systems.
The least critical cards are PCI-bus cards. The PCI BIOS normally takes care of assigning a good MemBase, and you should never have to deal with all the mumbo-jumbo below.
If the server gets it wrong, you may end up with a severe system crash (e.g. if it maps the video memory right on top of your system memory). If this happens, RESET IMMEDIATELY. Do not try to shut down cleanly, because the X-server, thinking it writes to the VGA memory, will write to system memory instead, and you'll be writing corrupted data to disk. If you did a sync prior to starting the server, there will be no harm done (only a filesystem check which should end up clean). DO NOT attempt to redirect the server output to a file on the system you're testing on (that will write data after you synced).
These are worst-case scenarios, and it is very unlikely this will happen to you. The text above is to make sure you are properly prepared, so that nothing serious happens.
When the server can't find a working linear memory base, it's time to experiment. The rest of this section deals with that.
Choosing a suitable MemBase can be quite tricky. If you have no way of determining the MemBase your card uses, trying to put it a few Mb above the system memory is a good first guess. E.g. if you have 16 Mb of RAM, defining MemBase 0x01000000 (=16M) or 0x01400000 (=20M) may work.
However, this may only work on non-PCI systems, as PCI systems mostly map all hardware above the 2GB mark. But then again, on PCI systems the server is almost always able to detect the correct linear memory base address. The only exception are those systems with more than one PCI VGA card.
On most VESA local bus (VLB) boards, there is an additional problem with address decoding. Most motherboards only decode the first 32, 64 or 128 MB of address space (a good pointer is to check the amount of DRAM that can be installed on the board: it will at least decode as much address space as it supports DRAM).
On such boards, you MUST specify a MemBase inside that range, or the actual address may wrap back onto system memory: if your system only decodes 128MB of addresses, and you set the MemBase to 128 MB, it will actually be decoded as being on address 0, which is probably exactly where your kernel memory is located. That is why the general guideline of putting the MemBase just above the system memory is a sound one: it stands most chance of actually being inside the decoded address range of the board. Unless your motherboard's entire memory space is filled with RAM.
If you don't know how much memory address space your motherboard decodes (and who does?), try using a "non-trivial" address, like 0x1FC00000, which has enough bits set to "1" to work on any motherboard, even if a few are not decoded. Keep in mind that using for example 0x10000000 may end up right on top of your system memory if the motherboard doesn't decode all upper address bits. You will only do that once.
Some other VLB boards can only map the linear framebuffer above the 1GB mark (0x80000000 and up), so you must use a MemBase that is higher or equal to 0x80000000.
Some other VLB boards can only map the linear framebuffer BELOW the 16 MB mark. So you may want to try booting your system with up to 12 MB of memory (some operating systems allow you to supply a boot-time parameter that limits the memory to a certain amount, so you don't have to open your computer to try this), and set the MemBase to 0x00C00000 (=12M).
Unfortunately, there is no easy way to tell what system you have (these details are mostly not in the motherboard manuals). Trial and error is the only road to success here. The server code will provide a default that works on most boards... but yours won't be one of those, of course.
There are some limits as to where the linear memory base may be put. On any ET4000W32, it must have a 4MB granularity (i.e. it can be put at 16M or at 20M, but not at 18M). On ET6000, it needs a 16M granularity (note: the ET6000 driver should be able to determine the linear memory base automatically, so you should never need to define MemBase in the first place).
On ET4000W32i, things are worse: the linear address base is hardwired on the card, and there is no reliable way to read it back from the card. You need to know the address in some way, and specify it. The current code does an intelligent guess at it, but this is no guarantee.
On ISA cards, things are much more simple: ISA only uses 24 address lines, and hence the linear memory MUST lie within the 16 MB boundary. Together with the 4MB granularity of the linear memory base address on ET4000 cards, this means that you cannot have more than 12 MB of system memory in the machine if you want to use linear memory. Hence, the only realistic MemBase for ISA cards is 0x00C00000. This is also what the server will automatically choose if it detects an ISA W32 card.
WARNING: you must not have over 12 MB of system memory in this case. Or if you have it, you must disable access to all memory above the 12 MB mark. Some operating systems allow you to specify at startup how much memory it is allowed to use, so you don't have to unplug some memory each time you want to use linear memory.
Yes, and in that case, you're out of luck.
There can be at least two reasons for this.
The first is the most common: the board manufacturer has left out the necessary connections and hardware to be able to use linear addressing. This means that no coding effort on this planet can help you with your problem: it is physically impossible to use linear addressing.
The second reason is that the current XFree86 Tseng linear addressing code is incompatible with the way your board is designed. The XFree86 Tseng code assumes a 1:1 mapping of the address lines from the bus (either ISA, VLB or PCI) to the address lines on the Tseng VGA chip. As unlikely as it may sound, this may NOT be the case!
Some very rare boards do not have such a 1:1 mapping (e.g. two address lines swapped). It is possible to support this type of hardware, but at this moment, this has not been implemented yet.
Other boards use external address decoding hardware that combines a number of address lines on the bus to a (smaller) number of address lines to the VGA chip. One such board for example uses three NOR gates (one 74F02 chip) to combine the 6 upper address lines to three address pins on the W32i chip. Obviously, this represents a 2:1 mapping, and not a 1:1 mapping. Therefor, this board is not "compatible" with the way XFree86 implements linear mode.
Simple: nothing works, or your machine locks solid, or it crashes, or a zillion of other things.
However, sometimes it is not always as obvious. Sometimes nothing bad happens: you just get a black screen, or a screen with rubbish on it, but nothing is drawn on it. Sometimes you get a core dump when the first application starts.
If acceleration is enabled in those cases, you will almost always see multiple "WAIT_ACL: timeout" messages in the server output. That is because the accelerator registers are also mapped in the linear memory, and if linear memory doesn't work, then also the accelerator doesn't work.
NOTE however that a WAIT_ACL message doesn't necessarily mean the linear memory address is bad. There are a number of other reasons for this message as well. But if you never saw these messages at 8bpp banked, then there's a good chance you have a linear memory problem ("banked" is the opposite of "linear", and is the default mode when "option linear" is not in the XF86Config file).
XF86Config file