|
|
Custom Linux: A Porting GuidePorting LinuxPPC to a Custom SBCShie ErlichRafi Yanai - my partner in the porting process Avi Rubenbach - without whom this wouldn't be possible
Chapter 1. Introduction1.1. Who needs to read this ?This guide describes a work in progress, to port Linux to a custom PowerPC-based board. This means making the operating system work on unfamiliar hardware. Anyone who is on the same track might benefit from reading this paper, as it highlights the pitfalls and problematic points along the way. 1.2. What do I need to know (why so much) ?Before attempting to port Linux, know at least the following: (whenever possible, a link to a proper information source is attached)
1.3. The toolsThis section describes the tools we used during the process. Most are trivial to install and use. When neccesary, consult the appropriate url or manual.
1.4. The hardwareThe board is based on PPC750 (PowerPC) processor. It is 6U VME64 standard. The board is designed to host two PCI Mezzanine cards (CCPMC) - Mezzanine cards that comply with Std CCPMC1386 can be attached.
1.5. Copyright & LicenseCopyright (c) 2002 Shie Elrich Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in Appendix A. Chapter 2. Bootcamp: How To Begin ?2.1. Creating a development environmentThe minimum requirments are obviously a development station and a target. However, the recommended way of working is having a third host which acts as a server. The server runs several services such as ftp, telnet, NFS, tftp (if needed) and CVS. The main role of the server is to run CVS and track version control, however once you can boot the target from network, the server will also hold the target images, and filesystem, which makes development much easier. Regardless, the first step is to install a tool-chain (compiler, linker etc.) for your target. The HardHat Linux cdrom includes all the needed files, and the installation sequence is documented in the HardHat Linux documentation. During the installation, you must select your LSP (basic software for the selected board), and HardHat will install a set of tools and a kernel source tree matching your LSP. We had a board that had vxWorks running on it, so we setup the target to boot using the standard vxWorks loader. Once the loader initiated, we used visionICE to take-over the target (so that vxWorks won't load an image file) and load a Linux image into the target. What you need to do at this point is get an ICE, connect it to the network and to the target - through a JTAG connection - and install the ICE software on your host. What should have been done so far:
2.2. Compiling the first kernelIf you've installed the Linux kernel that comes with HardHat, then cross-compiling should already be enabled in the kernel Makefile. If your kernel is not from the HardHat CD, you should enable cross-compiling in the Makefile by defining a CROSS_COMPILE entry in the following manner: (a code segment from the main Makefile)
The Linux kernel is modular, and allows you to configure it and choose which "blocks" should be compiled with the kernel. In order to do this, first cd /usr/src/linux (assuming your kernel source code is installed at /usr/src/linux). Once there, type make xconfig.After saving your options, you should make vmlinux to create a kernel image suitably for using with VisionICE. We will not go into more details here, as it's outside the scope of this document. For more information, try http://www.tldp.org/HOWTO/Kernel-HOWTO.html 2.3. Booting the machineFirst, configure the terminal program, in our case minicom, the following way: 9600 bps, 8 bits, no parity, 1 stop bit and no flow control of any kind. The serial port in Linux should be /dev/ttyS0 for COM1, /dev/ttyS1 for COM2 etc. Start the target. You should see the vxWorks bootloader on your terminal screen, and should be able to stop the boot sequence by pressing the space bar.
Once the target is stopped, run the VisionICE software and perform the following steps:
Chapter 3. Booting In The Dark3.1. Debugging with print_str()As stated in the previous chapter, the machine starts to boot, but nothing happens. At least, nothing that we can see. The screen is blank and no kernel messages appear. At this point, you have to ask yourself, is it really booting? Since the console wouldn't start, and ICE died real fast, we had no choice. We had to debug somehow, and the oldest way is good here - printing to the screen. Obviously, we couldn't use printk(), so we wrote a short function which pushes characters straight into the serial port. We used the boot process "map" shown in the previous section, and inserted some prints along the way. This helped us to know at what stage we are completing and where we're dying. The following piece of code prints a single character to the serial port, by polling it and waiting for it to be free.
3.2. Modifying code using compiler flagsAlthough it is not a porting issue, the way you modify your code matters. It's easier if you do it right the first time. The Linux kernel uses standard configuration flags CONFIG_XXXX (like CONFIG_PPC, CONFIG_ISA etc), which are used to mark a certain machine, architecture or device. We defined ourselves a new flag (let's call it CONFIG_TESTMACH), and surrounded our new/modified code with these flags:
3.3. Getting the console to work3.3.1. Forcing the kernel to boot our-wayOnce we discovered the kernel was indeed booting, but the console wasn't printing, it was time to begin. First, we forced the kernel to boot using a specified configuration for the serial port, in our case 9600n1, and did not allow any command line options or boot time considerations etc. The first place to go is drivers/char/tty_io.c, to console_init(). This function determines the console configuration at startup. Here's a small part of it:
Disappointed but not discouraged, we remembered that we didn't have a bootloader yet, and that we didn't really know if any option was being passed on to the kernel. "Maybe the kernel gets some garbage for command line?" we (again, naively) thought. So we tried to stop the kernel from parsing command-line options, and manually inserted our command line. This didn't help us much ;-) 3.3.2. Non-standard hardware - just say no!At that point, we didn't have a console, but we had time. So we dove a bit deeper into the console issues. Looking at drivers/char/serial.c, we came across serial_console_setup(). This function, apart from parsing command-line options, also configures the serial port by writing directly to it. Our hardware people decided it was a good time to let us know that our serial port wasn't standard. The lines that are used for flow control were not connected. We decided to remark-out the following line, which sets the RTS and DTR lines high, because we just didn't have them.
3.3.3. Let there be light: calculating baud rateFinally, we decided to check the baudrate. Did Linux mean what we thought it meant when it said 9600? Possibly not, since we didn't know how it computed that value. We've noticed that the file(s) include/asm-ppc/pmppc_serial.h (replace pmppc with your board name) included a definition of BAUDBASE, which is later used for everything regarding serial ports. It was computed using the board's local bus frequency, bus clock to system clock ratio etc. This seemed wrong, so we checked out what the base baud was in a vxWorks system we had running on the board, and changed it to:
Chapter 4. Linux Still Isn't Booting4.1. Memory probing, RTC and decrementorsNow that the console was working, we could see the real problems. The system wasn't booting yet. Since we were working with C code, we traced the code, and found that a function called sdram_size() wasn't completing correctly. The function probed a register for the size of the RAM, a register our board doesn't have. We made the function return a given value of 128MB, it's an ugly hack, but our board doesn't have a way of knowing the amount of RAM. Wwe had the same problems with a bunch of functions called todc_XXXX, mainly todc_get_rtc_time(), todc_set_rtc_time(), and time_init() since we don't have a RTC (real-time clock) chip on our board, and those functions were using it. For the time being, we made the todc_XXX function only set and get a constant date and time, since our board doesn't have a bios battery and so cannot keep time when powered off. Once all this was done, we found todc_calibrate_descr(), which again uses the RTC chip. We had to replace that function with our own:
4.2. Big-little endian (we should have known)4.2.1. Probing the CPC700Finally, we reached the PCI-probing part of the boot process, only to discover that it didn't work. We tried communicating with the CPC700 using cpc700_read_local_pci_cfgb(), which was supplied along with the PMPPC's LSP, and tried to read CPC's config register. We should have gotten 0x1014, which is the vendor ID, but we didn't. We realized that we were talking little-endian and the CPC was listening in big-endian. We made a small patch to the functions, so that we spoke big-endian to the CPC700. We could then read the vendor ID correctly, but the rest of it still didn't work. We didn't want to alter the code so that everything would be done big-endian style. 4.2.2. Making CPC700 speak little-endianWe discovered that the CPC700 can be initialized to do automatic byte-swapping, which does little-to-big endian convertion on the fly. As it seems, our board was initialized to do just that. We added a small code segment in setup_arch(), which checks if byte-swapping is enabled, and if so, disables it:
4.3. Ethernet: our first PCI deviceOur board uses an Intel ethernet chip, called i82559er, which has a module called eepro100. After compiling the module and booting, we discovered that the module isn't working, although an ethernet device was found. We guessed that it was an irq problem, and that the devices don't get the IRQs they need. We modified a function called pmppc_map_irq() to map our ethernet devices:
The next problem was that the module couldn't decide on a MAC address for the device. The MAC address should be written on an EEPROM chip (connected to the device), but we discovered that the hardware guys decided that i82559 doesn't need the EEPROM, so they removed it. After hardcoding a MAC address inside eepro100.c, the ethernet device finally worked. The final solution was to make the module read the MAC address from NVRAM memory, and if no other choice was available, to fall back to a default MAC address.
4.4. Some Miscellaneous IssuesWe had new problems, some would say good problems. We didn't have a bootloader yet, however we needed to pass a command line to the kernel at boot time. We hard-coded the command line into the kernel inside the parse_options(). After that was finished, we made console_init() and serial_console_setup() work the way they should. They no longer ignored the command line, but still RTS and DTR stay low. Another important issue was memory mapping. The file arch/ppc/mm/init.c contains a function called MMU_init(). This function is actually a big switch statment, divided by the machine type. Each machine maps its memory using the setbat() and ioremap() functions. The BAT mechanism is a way of translating virtual addresses into physical ones. Thus, setbat() is used by specifying a virtual address, a physical address and a page size. Not every size can be used here; you should use one of the finite set of sizes, ranging from 128KB to 256MB. We mapped our IO memory so that virtual equalled physical. As mentioned, there is another way of mapping memory - ioremap(). ioremap() is used to map physical addresses into virtual ones, making them available to the kernel. The function does not allocate any memory, simply returns a virtual address by which one can access the memory region. The following is a snippet from MMU_init():
Chapter 5. Linux Is Booting ... What Now ?5.1. The 64 bit barrierThe CPC700 has a "feature" which is supposed to make some memory access use 64 bit wide. This is a problem since some test-and-set registers on our board might get set unintentionally, because we were trying to read something 16 bits lower. In order to solve this situation, we set the memory controller to 64 bit wide intervals. If you try to access those areas in another manner (8 or 16 bit access), the CPC700 simply throws them away. We had to be able to read/write those areas, since important "discretes" (controlled by an Altera device) were mapped there. In order to access those areas, we needed a function that does a 64 bit write. As far as I know, doing a 64 bit write on a PowerPC is possible in two ways: using cache lines and using a floating point register. The floating point register is a 64 bit sized register, so when we write it, the whole 64 bit get written. The problem is that you can't do floating point in the kernel. Since the kernel doesn't save the floating point registers during context switch, it doesn't allow FP, and will throw an exception if done in the kernel. After messing with cache lines, we decided to go the FP way, and added the following function:
5.2. Booting from flashWhile Linux was booting using an NFS filesystem, this was not enough. For an actual field product, we needed Linux to boot from an independent device, without the need for a network at all. We decided to create a special kind of image, called initrd, which is basically a Linux kernel with a compressed file. The compressed file includes a Linux filesystem. The filesystem is unpacked to a ramdisk on boot, and mounted as the root filesystem. During the boot process, the bootloader relocated the kernel image to address zero - which was fine, and the initrd part to a higher address. The area to which initrd was relocated was not mapped in our kernel's memory, and all we got was a kernel error (access to bad area). After modifying the bootloader to relocate initrd to a different address, all was fine and Linux booted succesfully.
Appendix A. GNU Free Documentation LicenseVersion 1.1, March 2000
A.1. PREAMBLEThe purpose of this License is to make a manual, textbook, or other written document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others. This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software. We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference. A.2. APPLICABILITY AND DEFINITIONSThis License applies to any manual or other work that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language. A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (For example, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them. The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, whose contents can be viewed and edited directly and straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup has been designed to thwart or discourage subsequent modification by readers is not Transparent. A copy that is not "Transparent" is called "Opaque". Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML designed for human modification. Opaque formats include PostScript, PDF, proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML produced by some word processors for output purposes only. The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text. A.3. VERBATIM COPYINGYou may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3. You may also lend copies, under the same conditions stated above, and you may publicly display copies. A.4. COPYING IN QUANTITYIf you publish printed copies of the Document numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects. If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages. If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a publicly-accessible computer-network location containing a complete Transparent copy of the Document, free of added material, which the general network-using public has access to download anonymously at no charge using public-standard network protocols. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public. It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document. A.5. MODIFICATIONSYou may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:
If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles. You may add a section entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard. You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one. The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version. A.6. COMBINING DOCUMENTSYou may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice. The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work. In the combination, you must combine any sections entitled "History" in the various original documents, forming one section entitled "History"; likewise combine any sections entitled "Acknowledgements", and any sections entitled "Dedications". You must delete all sections entitled "Endorsements." A.7. COLLECTIONS OF DOCUMENTSYou may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects. You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document. A.8. AGGREGATION WITH INDEPENDENT WORKSA compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, does not as a whole count as a Modified Version of the Document, provided no compilation copyright is claimed for the compilation. Such a compilation is called an "aggregate", and this License does not apply to the other self-contained works thus compiled with the Document, on account of their being thus compiled, if they are not themselves derivative works of the Document. If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one quarter of the entire aggregate, the Document's Cover Texts may be placed on covers that surround only the Document within the aggregate. Otherwise they must appear on covers around the whole aggregate. A.9. TRANSLATIONTranslation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License provided that you also include the original English version of this License. In case of a disagreement between the translation and the original English version of this License, the original English version will prevail. A.10. TERMINATIONYou may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. A.11. FUTURE REVISIONS OF THIS LICENSEThe Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/. Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation. A.12. How to use this License for your documentsTo use this License in a document you have written, include a copy of the License in the document and put the following copyright and license notices just after the title page:
If you have no Invariant Sections, write "with no Invariant Sections" instead of saying which ones are invariant. If you have no Front-Cover Texts, write "no Front-Cover Texts" instead of "Front-Cover Texts being LIST"; likewise for Back-Cover Texts. If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software. Appendix B. TrademarksLinux® is a registered trademark of Linus Torvalds. MontaVista™ is a trademark of MontaVista Software Inc. PowerPC® is a registered trademark of IBM Corporation. Windows® is a registered trademark of Microsoft Corporation. vxWorks™ and Vision ICE™ are trademarks of Wind River Systems Inc. ProCOMM™ is a trademark of Symantec Corporation. |
|||||||||||||||||||||||||||||||||||||||||||||||
