T|X internal storage

TX internal storage is a NAND flash chip. This is difficult for a few reasons, mostly, because NAND needs wear-leveling, and special filesystems, neither of which is standardized. There isn't even a standard way to lay out a partition table! Thus to get TX internal storage to work there are three components to be created: BlockDevice driver to talk to the NAND chip, PartitionManager plugin to figure out the partitions on the device, and filesystem driver for the FS used in the TX (something called "BFFS").

I am happy to report that the first two pats are complete and working. The third is harder, however I've had some help from a programmer in China (irc handle tech_junkie_chz_5481), who seems to know a lot about this BFFS and has helped me craft a driver for it all in one night. The testing is going to start soon.


PS: Hello from Buenos Aires!

UiLib

Did some work on UiLib and various UI components.

Now off to a 2-week vacation :)

E2 final update

T|E2 now boot reliably into DGOS. Now i will go back to userspace work.

T|E2

T|E2's processor is a generation older then TX's and LD's (and htu slacks some features DGOS kernel uses), but the kernel comes up (somewhat) on it. Will work today on making that 100%

TX NAND further updates

TX NAND speaks to me just fine, AND i am able to parse palm's strange partition table, ECC codes, as well as bad block chains. Next: parsing their filesystem used on the TX: It's called "blunk flash file system" and getting an official copy of the spec for it costs $15,000

TX NAND speaketh...

...and it says CE F1 80 15



[for those who have no idea what that means - the TX internal storage driver had been started, and just successfully spoke to the flash chip. Those numbers are the chip's ID number]

Update

My new job is getting me into habit of daily status updates, so here goes :).

LD usb driver still being stubborn, TX internal memory chip can be read/written/erased, but data is still somewhat cryptic, next demo will demonstrate a functionality like CardExport, but under DGOS on an LD, and the one after that will be a TX demo. Lots of work needs to be done on the UI, and I welcome all takers, who want to make an effort to develop/help with the UI design/programming (I am just one man, working an average of 45 minutes a day on this, afterall)

Small update

A small but important bug was fixed in module makefiles, and lifedrive USB driver has been split off into its own module, and some more work was done on it. now it's maybe 60% done :)

Look for more updates soon.

GfxLib transparency works well

A small video. Shows two semitransparent layers interacting using GfxLib. Also shown is the boot process. Enjoy.






The quickly-written and messy source code to this simple test app is as follows:

#include <kTypes.h>
#include <errors.h>
#include <RtLib.h>
#include <GfxLib.h>



static char __x[128];
#define printf(...)   do{sprintf(__x,__VA_ARGS__);_DGOS_DEBUG_TMP_printStr(__x);}while(0)
#define perr(...)   do{printf(__VA_ARGS__); while(1) SchedYield();}while(0)

int _start(){
   
   GfxImage* i;
   GfxLayer* L;
   GfxLayer* L2;
   UInt32 iw, ih, sw = 324, sh = 484;
   Int32 x = 0, y = 0, dx = 1, dy = 1;
   UInt16* d;
   Err e;
   
   printf("TEST RUNNING\n");
   
   if(!GfxLayersInit(GFX_LIB_ROTATION_NORMAL)){
      perr("cannot init GfxLayers\n");
   }
   
   printf("setting color\n");
   GfxLayersSetBackground(GfxColorMakeRGB(192,255,192));
   
   printf("creating layer\n");
   L2 = GfxLayerCreate(0, 0, 324, 484);
   if(!L2) perr("cannot create layer with bg\n");

   
   printf("calculating layer\n");
   printf("L=0x%08x\n", L2);
   printf("I=0x%08x\n", GfxLayerGetImage(L2));
   printf("D=0x%08x\n", GfxImageGetData(GfxLayerGetImage(L2)));
   printf("A=0x%08x\n", GfxImageGetAlpha(GfxLayerGetImage(L2)));
   
   d = GfxImageGetData(GfxLayerGetImage(L2));
   MemSet(GfxImageGetAlpha(GfxLayerGetImage(L2)), 324*484, 0xFF);

   for(y = 0; y < 484; y++){
      for(x = 0; x < 324; x++){
         UInt32 r,g,b;
         
         r = y * 256 / 484;
         g = y * 256 / 484;
         b = y * 256 / 484;
         
         r = ((r * x) + (256 * (323 - x))) / 323;
         g = ((256 * x) + (g * (323 - x))) / 323;
         
         
         *d++ = (((y / 10) & 1) == ((x / 10) & 1)) ? \
                  GfxColorMakeRGB(r,g,b) : \
                  GfxColorMakeRGB(0,0,b);
      }
   }
   
   printf("drawing layer\n");
   GfxLayerRedraw(L2);
   
   
   printf("loading img\n");
   
   e = GfxLibLoadPng(3,L"/test.png", &i, false);
   if(e) perr("cannot load image: 0x%04x\n",e);
   
   GfxImageGetDimensions(i, &iw, &ih);

   printf("img is %dx%d\n",iw,ih);
   
   L = GfxLayerCreateWithImage(0, 0, i);
   if(!L) perr("cannot create layer with image\n");


   x = 0;
   y = 0;

   while(1){
      
      x += dx;
      y += dy;
      
      if((x + iw >= sw) || x <0){
         dx = -dx;
         x += 2 * dx;
      }
      if((y + ih >= sh) || y < 0){
         dy = -dy;
         y += 2 * dy;
      }
      
      GfxLayerSetLocation(L, x, y);
   }

   while(1) SchedYield();   //idle well
}

Demo Time?

Perhaps it's time for a small demo? Perhaps a video of the boot process on an LD or a TX? Or maybe a small demo of a simple doodling app. While hardly impressive by itself, both of these build on an entirely new OS, which makes them impressive.

Or perhaps wait till the UI lib is finished, and a new semi-transparent UI can be demoed as well?

Or perhaps both?

What do you think. Comment to reply.


[currently LD is the most supported device, with TX close behind given that it is so similar. T|E2 was tested a few times and worked reasonably well, to some extent, but lately it doesn't. I think my new scheduler code is to blame. I'll look inot it later.]

on GfxLib and alpha-blending

GfxLib has seen some new code added to it. This is the general graphics library for the OS, providing basic drawing (like lines and rectangles), as well as advanced drawing (alpha-blended images). The alpha-blending code exists in multiple versions ,to take advantage of different processors. The "generic C" version can pull off 15 frames per second on alpha-blending a fullscreen image on 324x484 resolution. This version will run on any ARM processor ever used in Palm devices. The "fast" version requires a ARMv5 processor (like one in iQue, Zodiac) and can pull off 21 frames per second on the same fullscreen alpha-blended image. The "WMMX" version requires an XScale CPU (like one in LifeDrive, TX) and uses Intel's WMMX (their MMX for mobile) and is twice as fast as the "genereic C" version, managing a whopping 30 frames per second on fullscreen alpha-blending test.

These FPS numbers may seem low, but remember that (1) it is rare to redraw the fullscreen, generally only the changed area needs redrawing, (2) these might become faster as I optimize them more (3) comparing to more modern hardware is pointless, since almost all modern hardware has graphics acceleration chips and PalmOS devices do not (with the exception of the Zodiac, but I am not supporting it anyways for now)



For the technically curious: C version processes one pixel at a time. ARMv5 version processes two pixels at a time, using the "Enhanced DSP instructions" that this architecture provides (SMUL<x><y> and SMLA<x><y>) to do the parallel multiplication. The WMMX version processes four pixels at a time and makes heavy use of WMMX instructions (TBCST, WLDR, WSTR, WUNPACKEL, WSUB, WAND, WOR, WSLL, WSLR, WMUL, WMAC) to do parallel processing of al four pixels.

Also a rather large (9%) speedup was attained by strategically placed "PLD" instructions, that instruct the D-cache to pre-load data into the cache.

PalmOS-based loader

PalmOS-based loader has been rewritten and now is finished. It loads DGOS as if it was just another PalmOS app. You tap it, it evicts PalmOS from ram and loads the DGOS kernel in under a second. It even has a [somewhat] cute icon:

Simplicity

Here is a simple kernel module, demonstrating how easy it is to write and build one.


kmod.c

#include <kernel.h>
#include <errors.h>
#include <timers.h>
#include <usrProcess.h>





Err kmod_main(UInt32 reason, void* data){
 
 UInt64 time = timersGetTicks64();
 
 if (reason == K_MOD_LOADED){
  
  charsPrintF("Module loaded at 0x%08lx\n", time);
 }
 else if(reason == K_MOD_UNLOADED){
  
  charsPrintF("Module unloaded at 0x%08lx\n", time);
 }
 else{
  
  charsPrintF("Module called with unknown command (0x%x,0x%08x)\n", reason, data); 
 }
 
 return errNone;
}

Makefile

LIBNAME = kmod.kmod
CC = arm-linux-elf-gcc
FLAGS = -O4 -fpic  -fshort-wchar -mthumb-interwork
SLIBS = 
DLIBS = ../KernelLib/KernelLib.lib
CCFLAGS = $(FLAGS) -I../Src/public -I../Src
LDFLAGS = $(FLAGS) -nostartfiles -nodefaultlibs -nostdlib $(DLIBS) $(SLIBS) \
          -Wl,-shared -Wl,-soname,$(LIBNAME)




kmod.kmod: kmod.o
 $(CC) kmod.o -o kmod.kmod $(LDFLAGS) -e kmod_main

kmod.o: kmod.c
 $(CC) $(CCFLAGS) -o kmod.o -c kmod.c

clean:
 rm -rf *.o kmod.kmod

funny code fragments

Err elfUnload(Process* p, void* base){
 
 if(p == taskGetKernelProcess()) {
  
  Panic("ELF unload in kernel not implemented. why are you trying it anyways?");
 }

Networking

Wifi chipset in lifedrive and tx is not documented. i do not plan to reverse engineeer it. I plan to run the palmos driver in a palmos container layer, and use it to talk to the wifi chip. palmos supports one raw socket per interface, thus giving me all the network access i need, and the DGOS networking stack can then use that interface to talk to the wifi chip directly.

initfs

When DGOS boots, it has no filesystem or device drivers. This is a problem since the filesystem and block device drivers are modules - files on the filesystem. We are thus left with a chicken-and-egg scenario that we need the filesystem access to load modules and we need the filesystem modules to have filesystem access. This is solved using initfs. This is a simple flat filesystem in ram that has just the bare minimum modules required to bring up the filesystem and device support needed to load the rest of the system. The bootloader loads two files to RAM. The kernel and the initfs image. The kernel then uses the initfs image to load the modules needed to access the main filesystem. initfs memory is then freed, and with this new ability to read the filesystem the kernel can go on booting.

DGOS used to have a very simple initfs FS and init_blk block device in it. I decided to replace initfs with just FAT, since it is simple, and there is little harm in compiling it into the kernel. This also means that the initfs image for most devices will now contain just one module - the block device driver for SD card (or in the case of lifedrive - two: SD and HDD drivers). FAT driver will now be part of the kernel and thus doesn ot need to be a module in initfs.

Artwork [or some likeness thereof]

FAT32?

done

Project is alive

I've been having lots of personal things to deal with lately, hence no updates here, but project is moving forward :)

FAT32.n+1

Busy at work lately, so had little time to work on the FAT code, but the dir code now works well, and file code is almost done as well...

FAT32 again

FAT32 work is still going on. Yes it's a pain in the ass, but it's moving. LFN code is written. FAT code is written. Just file and dir code left

FAT32

FAT32 is a terrible filesystem. Fat16 and 12 are ok because the FAT is small enough to fit in memory (max 128K) so that it can be fast. FAT32 FAT can be up to 256 MB in size, and cannot be fit in memory. Since FAT32 does not feature a free cluster bitmap, finding free clusters on a moderately-filled FS can be a big pain. My FAT32 driver attempts to optimize this by creating a compressed free-cluster bitmap in memory and fills it lazily as FAT sectors are read for general use. This means that the longer one uses a FAT32 volume, the faster file creation and writes get. This is a strict improvement over most other FAT32 drivers, which start slow ad remain slow.

General Project Status

I get asked a lot what the project status is, and when i can release a beta. First let me say that this question is rather pointless. No, not because I want to be cruel and keep it a secret. It's because I really do not know and have no way of knowing. I work on this in my free time, so while I can estimate the percentage completion of the project, I cannot estimate the time remaining because I do not have any hard time commitment to this project.

That being said, it is also worthwhile to point out that normally a whole modern OS (even a simple one) is work for hundreds of engineers and a year or two of work, while I am just one guy and working in my free time.

---

The heart of any OS is the kernel. It provides memory, scheduling, storage, and arbitration of other resources. DGOS kernel is written from the ground up by me over the last few months. IT provides full protected-memory processes with multitasking and support for multiple hardware storage and filesystem drivers. It has a plugin architecture, which functions in a way not unlike the way microsoft windows drivers work. The kernel works, and I do not forsee much more work in it coming up, since all new things will be loadable kernel plugins, and not work on the kernel itself. The kernel is written in C and arm assembly and is compiled by the metrowerks codewarrior compiler. All plugins have to be compiled by GCC. (There is no support in CodeWarrior to produce working plugins, though with enough effort, it can be done)

Close to the kernel are the drivers. They provide various services and methods to work with hardware. DGOS kernel currently best supports the PXA270 processor (such as the one in a Palm LifeDrive). There are drivers for timers, real time clock, LCD, sound and touchscreen, the notification LEDs, the built-in hard disk, and the SD card. This, of course, leaves out WiFi, Bluetooth, and USB. Work on USB is ongoing, and WiFi and bluetooth will be done later or provided by PalmOS drivers wrapped in special DGOS wrappers (not unlike ndiswrapper does on linux). Currently written drivers are in the kernel, all future ones will be loadable plugins.

Above the kernel sits RtLib. It's the userspace library that wraps the usually-strangely-called system calls into ARM-EABI compatible function calls, as well as providing basinc runtime services like heap management. You can read all about it in a recent post about RtLib => [here] <= . RtLib is still being written, but it already provides many useable things, and simple apps have run on top of it.

Userspace apps run on top of RtLib and may use any number of other libraries, as they wish. Currently two uch libraries exist. The DGOS port of jpeglib (which is interesting in that it required no code changes and built form the original source just fine, and ran) and GfxLib (which is a simple library that currently provides abilities to draw rectangles, lines, circles, and a very very simple fixed-width font). In the future it will probably be expanded slightly, but in general it's expected to be the basic simple drawing library.

The UI will be provided by UiLib, which still needs to be written. Luckily this is not a huge effort, since a lot of SkinUI code can (and will) be reused for it.

PalmOS compatibility will be provided by an application whose job will be to

• Provide a full R9-table of functions that NAtive PalmOS apps need
• Host such apps and handle their hardware accesses in appropriate ways
• Parse and access PalmOS's data structures on disk, so that PalmOS and DGOS can share PalmOS files(reboot into PalmOS and your datebook has the entries you just added in DGOS
• Allow Palm's PACE to run, or implement DGOS's own version of PACE
• Provide access to things like network and expansion cards

Of course no OS is complete without its own userspace apps, and an SDK. The SDK is hard since putting one together is a monumental task. Luckily for DGOS this is not hard, since almost any linux library will run on DGOS is recompiled for it. Of course the UI toolkit will be different, BUT to PalmOS programmers the UI apiw ill be familiar and simple, so the SDK will be a mix of linux standard APIs and PalmOS UI apis, with a mix-in of user-requested features I choose to implement.

So so summarize, the project is doing well, and there is a chance of a release of something of interest to developers (but likely not to final users yet) really soon.

FAT32

the fat32 library i had apparently is more worthless than i had imagined, and i had pointlessly spent time outfitting it with LFN support. Writing a new one now

On Architecture: RtLib

RtLib is the main runtime library that all apps will dynamically link against. While it is technically possible to write a DGOS app that does not use RtLib, it's needlessly complicated and pointless(One would have to do manual heap management, manually initialize all libraries, manually get and parse app params, manually make systemcalls, and not use any delay-loaded dynamic libraries). It provides useful things like

• Runtime functions needed for support of float and double types (no need for each app to have a local copy and waste memory on it)
• Systemcall wrappers (like VfsFileOpen or GetCurTid)
• Memory management (like malloc, free, and realloc)
• Other commonly useful things (like CreateThread or CreateProcess)
• Manual library management(RtLibLibLoad, RtLibLibUnload, RtLibLibFindSym)
• Syncronization primitives (like mutexes, semaphores, mailboxes, events, and request queues)
• Startup code (init libraries, init heap, allow libraries to self-initialize, get app command and parameter block from the kernel, find main(), call main() )

It is currently my intention to make RtLib open source, just like all other non-kernel pieces of the OS.
Just like all non-kernel components of the OS, RtLib is compiled using the arm-elf-gcc toolchain (I use version 4.2.3). As a sidenote, the kernel is compiled using CodeWarrior, but all loadable kernel modules will be written using GCC as well.

Work Continues

I'm back from my vacation, and work goes on. Now will be doing some RtLib work for app startup.

Module Loader.3

Unloading stuff works too :)

Module Loader.2

Just did some testing and bug fixing. Module loader works as expected. Here's some technical info.

Each process has a list of loaded modules in it, including a reference count. When module A depends on module B, and A is loaded B is loaded automatically. If it was already a loaded (perhaps an already-loaded module C depended on it), its reference count gets incremented. This all assures that when all modules depending on a certain module are unloaded, it can as well be unloaded, so as to free up the address space it takes up. The per-process module list is just a linked list, containing base address, module name (not path), and a pointer to the system-global list.

The system-global module list is what enables memory sharing of code and read-only data between processes loading the same module. It is a hashtable, indexed by a hash of the PATH of the module. This means that process A can load library A version X from path Y ad process B can load library A version Z from path T and they will not collide. The global table entries also have reference count, but this is per-process. This means that if 2 processes load module A, it's ref count in global table will be 2. If one process loads the module 5 times, the global reference count will still be 1. In any case the point is that when no more processes have a module loaded, we can evict it from memory, and free it up.

Module Loader

Module loader appears to be complete. It now shared code and rodata pages between processes, and r/w data is shared using COW pages. This significantly reduces use of physical memory and is thus good. Now work moves on to testing this, and then to pagefile code.

COW & memory sharing

COW(copy on write) memory has been implemented, and I am 90% of the way there to implement code & data sharing between processes that load the same modules (thus saving valuable physical memory)

Sector Cache

A simple sector cache has been added for block devices, causing a great increase in I/O throughput. :)

Debugging & USB

To help me debug userspace apps, the kernel has a builtin GDB stub, that allows debugging of any app on the device using the serial port. Unfortunately my PC has no serial port and I have no USBtoSerial adapters, so I am now writing the USB driver code, so that I can debug using USB.

This should be a useful tool for all other developers too, of course.

IT WORKS!!!

The kernel can now load libraries and executables, which can link dynamically to each other.

This works as expected! This is a huge step, and now work on RtLib resumes

ElfLoader works!

May the man who created the Arm version of the Elf standard burn in hell! But despite his best efforts at creating the most convoluted and impossible to load binary format in the world I have written a loader for it. So far the loader can load and relocate an executable and any shared libraries it needs, including function calls between them!

ELF.3

The elf loader is well on track, and the continuing work on it has helped uncover some bugs in the VAD code (which upon closer inspection were bugs in the AVL tree code that is the storage for the VAD tree)

ELF.2

After spending more than 20 hours trying to do a build of gcc crosscompiler. Nothing worked. After a LOT of work, i managed to build gcc/gdb for target arm-elf, host linux-i686.

Now continuing work on the DGOS elf loader...

ELF

I implementing an ELF loader for the kernel. This will allow usage of unmodified arm-elf-gcc to make DGOS apps, libraries, and kernel modules.

RtLib

RtLib got a lot of work done on it, it now loads to the init process, as expected, and heap management as well as thread management work well.

Block Devices, PartitionManager, Loadable FSs/BlockDeviceDrivers

Block device manager got a few fixes and improvements, including ability to notify many clients about addition/removal of devices.

Parititon manager was separated out from VFS code and is now a separate component. This makes it easier to use.

Block device drivers and filesystems can now be implemented as loadable kernel modules, which can make future support of other FSs and devices easier.

VADs, stack guards, signals, libraries

VADs are now working, this means that pagefaults can now be sorted based on cause and handled thusly. Thereforw non-committed stacks now work, stack guards work, and user processes are notified of pagefaults and are given a chance to handle them.

Also some more work was done on library support, and unloading now works too, as well ask keeping track of all such actions.

usrCallback

usrCallback module was just added to the kernel. It allows the kernel to call a function in a user app and get back some results, just like a normal callback, but while respecting all kernel-user boundaries.

DBF update2

RtLib is now a DBF library and loads well.

DBF format and loader

DBF is the binary format used by DGOS. It is simple, and almost any ARM compiler cna be made to emit it. It allows for libraries and executables that may or may not be compiler as position-independent code. All libraries and applications always have read-write access to their globals, and code is write-protected, as any good OS should so.

The kernel DBF loader code is now written and works.
RtLib work is proceeding.

Kernel fixes

A few kernel bugs were fixed (a serious kernel stack corruption by a stray DMA was a fun one to track down).

The SD/SDHC/SDXC/MMC driver in the kernel now works very well, and integrated with the block device manager and the FS manager gladly mounts volumes on it.

Now back to working on RtLib


[also the GDB stub in kernel is almost complete, so debugging RtLib and user apps will be nice and easy]

AC97.2

Touchscreen works very well using the new DMA driver for AC97

in fact it can even distinguish between stylus and finger press :)

AC97

Got AC97 to work with DMA, so now audio works, and touchscreen data is sent to the memory without CPU involvement :). Calibration works well, and the pressure-sensitivity of the panel is used too, so instead of the plain boring (X,Y) coordinates, DGOS provides (X,Y and Z) Where Z is the force user is applying to the touchscreen. It can be used for better flow in note-taking applications for example, or for sensing finger presses for gesture analysis [finger generates bigger Z values]

Sorry for the delays

Sorry for the delays

I will now dedicate a lot more time to DGOS. I also welcome any and all developers who would like to help with this effort. If you would like to help, please email me at dmitrygr@gmail.com

Updates

Had a lot less time lately thanks to my work and a new hobby (rc airplanes)

Some work was done still. VFS is working and stable. Init code is being made