Cross-compiling (ES series)
This is Work In Progress !!
No answers/solutions can be expected here yet. If you need quick answers Google the forums.
@developers:
Do not change or update this page without first checking/asking on the "Talk" page or
support forum thread for latest status/info.
This is an introduction to cross-compiling for MST10P (MStar/MediaTek) based TV
sets. It is primary intended as a crash course for getting even a novice to be
able to quickly compile his/her own programs to run on their TV sets. As such,
we will assume that you are using a Windows based PC with a basic installation
of Cygwin. The modification for using a Linux based PC will then be minimal and
even simpler.
Contents
- 1 Cross Compiling
- 2 (A) Installing the Cross-Compiler
- 3 (B) Determine the TV processor & architecture
- 4 (C) Compiling "HelloWorld"
- 5 (D) Installing the Samsung Kernel Sources
- 6 (E) Building the Kernel using Cygwin
- 7 (F) Compiling the Kernel
- 8 (G) Compiling a Kernel module
- 9 References
- 10 (Z) Place Holder Template
Cross Compiling
Introduction
If you have never cross-compiled anything before, this is the right place for you. If you already have experience and knowledge with cross compilation, this wiki may still be helpful to get you started. If you are looking for information on how to build your own cross-compiler, this place is not for you. (Look HERE instead.)
What this Wiki will cover and not.
- We will use a popular pre-compiled cross-compiler.
- We will work on a Windows (Intel/AMD) based PC.
- All examples herein are based on a UExxES5700 TV set, and should be reproducable on the same.
- We will not cover the compilation of a cross-compiler!
- We will not cover other cross-compilers, operating systems or processor architectures.
A few Questions and Answers:
Q: What is a cross-compiler?
A: Basically it is just a compiler built to run on one type of processor (e.g. Intel x86),
but which is built and configured for compiling code for another processor (e.g. ARM).
Q: Should I get a pre-compiled cross-compiler or compile my own?
A: We hate to waste our time compiling compilers, so always try to find a pre-compiled one!
Q: Where can I find help to configure my cross-compiler?
A: Not here. If it's not already in here or in our forums, we don't know.
Q: Do I need to install the TV specific platform sources? (Such as UExxES6xxx.zip ?)
A: ** It depends on what you need to compile. (See below.)
Q: Do I have to install the Samsung platform ARM toolchain? (Such as VDLinux-ARMv7-4.4-202-toolchain-v2r2-20110630.tgz ?)
A: ** Hopefully not, but it depends on what you need to compile. (See below.)
** = unknown and not fact!
Things you need to get started.
WIP! This need checking and adjustment...
1. Install Cygwin for Windows. (Not necessary, but very helpful for the various *nix tools and file utilities.) 2. Install a good text editor (EditPlus, Notepad++ etc.) 3. Download a suitable pre-compiled cross-compiler. 4. Download your TV kernel sources. 5. Download your TV firmware sources. (?) 6.
Steps:
(A) Install & Verify the pre-compiled cross-compiler on your PC.
(B) Verify your TV sets processor / architecture.
(C) Compile "HelloWorld" and run it on TV.
(D) Installing the Samsung Kernel Sources.
(E) Setting up your development environment.
- Setting up your PATH's + other shell/system variables)
- Setting up your Makefile
- other?
(F) Compiling a Kernel Module
(G) Compiling the Kernel
Extras:
(D)Installing the Samsung cross-compiler.(E) Installing your TV Kernel sources. (F) Installing your TV firmware sources. (G) Compiling the Kernel (H) Compiling a Kernel module
...
(A) Installing the Cross-Compiler
Go to the Mentor Graphics website, and download the "Sourcery CodeBench Lite Edition" from HERE.
(You'll need to supply an email to get a download link.)
There you will get a few different choices based on the platform. You will have choices such as:
arm-2012.09-63-arm-none-eabi.exe arm-2012.09-63-arm-none-eabi-i686-mingw32.tar.bz2 arm-2012.09-64-arm-none-linux-gnueabi.exe arm-2012.09-64-arm-none-linux-gnueabi-i686-mingw32.tar.bz2 arm-2012.09-64-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2
If you use a x86 Windows based system, choose: "arm-2012.09-64-arm-none-linux-gnueabi.exe".
Run it, and when asked, change the installation directory to something simple like: C:\zarm\csbench
The rest of the installation procedure is self explanatory.
After installation, verify that the cross-compiler PATH variable is properly set and working:
$ arm-none-linux-gnueabi-gcc --version arm-none-linux-gnueabi-gcc.exe (Sourcery CodeBench Lite 2012.09-64) 4.7.2 Copyright (C) 2012 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Since this is a Windows installer, Cygwin may or may not catch the updated system PATH variable. Open a new Cygwin shell and check:
$ echo $PATH
If it's not working, you'll have to add the following line in your ~/.bash_profile file.
(On some systems this file is called ".profile".)
PATH=${PATH}:/cygdrive/path/to/csbench/bin;
This should do it.
*******************************************************************************
If you want to have access to the cross-compiler's man pages you'll have to
add the following line to /etc/man.conf:
MANPATH_MAP /path/to/csbench/bin /path/to/csbench/share/doc/arm-arm-none-linux-gnueabi/man
and/or possibly this line to your ~/.bashrc
MANPATH=${MANPATH}:/path/to/csbench/share/doc/arm-arm-none-linux-gnueabi/man;
*******************************************************************************
If you have more questions, check out the Sourcery ARM FAQ.
(B) Determine the TV processor & architecture
In order to properly compile and run anything, you need to know what processor you're programming for. Here is how to find that information.
1. Root your TV and login to a shell. (Instructions HERE.)
2. Verify your TV's processor type and architecture by executing the following at the shell prompt.
shell> cat /proc/cpuinfo Processor : ARMv7 Processor rev 0 (v7l) processor : 0 BogoMIPS : 1794.04 processor : 1 BogoMIPS : 1794.04 Features : swp half thumb fastmult vfp edsp neon vfpv3 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x3 CPU part : 0xc09 CPU revision : 0 Hardware : amber3 Revision : 0000 Serial : 0000000000000000
Now, this is not obvious in any way. Problem is that an "ARM" processor is really
only a license to manufacture a processor according to ARM Holding's specifications.
Therefore any hardware manufacturer can have a processor with an ARM "core". In
addition, and to make it even more confusing, an ARM "core" belong to an ARM
"architecture" which belong to an ARM "family", whose numbers are little related...
The only way to get some insight is looking at the WikiPedia entries:
1. "List of ARM microprocessor cores"
2. "ARM architecture"
and compare what you find there with the interpretation of the ARM "Features" above, where:
----------------------------------------------------------------------------- Feature Description ----------------------------------------------------------------------------- swp SWP (SWaP) instruction, which is used to implement a binary semaphore (mutex) half Half-precision (16-bit) floating point, "__fp16" data type in gcc thumb Thumb instructions fastmult Fast multiplication vfp Vector Floating Point instruction extension edsp Enhanced DSP instructions neon NEON SIMD instructions vfpv3 Vector Floating Point instruction extension Version 3 -----------------------------------------------------------------------------
So from the above we think we have a dual core processor of the ARMv7-R or
ARMv7-A architecture. (Of either the Coretx-R or A families, respectively.)
But the only NEON enabled processors with both DSP and VFPV3, are the
Cortex-A8 and "Cortex-A9 MPCore". But this is not good enough...for a perfectionist.
So we search in the ARM on-line documentation for
"Main ID Register":
Main ID Register bit functions:
----------------------------------------------------------------------------- Bits Field Value Function ----------------------------------------------------------------------------- [31:24] Implementor 0x41 implementor: ARM [23:20] Variant 0x3 variant number or major revision of the processor [19:16] Architecture 0x7 architecture is given in the feature registers [15:4] Primary part number 0xC09 part number: Cortex-A9 [3:0] Revision 0x0 revision number or minor revision of the processor -----------------------------------------------------------------------------
Here is a brief list of ARM primary part numbers.
ARM core CPU part ------------------------ ARM920 0x920 ARM926 0x926 ARM946 0x946 ARM966 0x966 ARM1136 0xb36 ARM1156 0xb56 ARM1176 0xb76 ARM11 MPCore 0xb02 Cortex A5 0xc05 Cortex A8 0xc08 Cortex A9 0xc09 Cortex A15 0xc0f Cortex R4 0xc14 Cortex R5 0xc15 ------------------------
We finally conclude that our TV processor contains a dual core Cortex-A9 MPCore from the ARMv7-A architecture.
Done!
(C) Compiling "HelloWorld"
We would like to compile our "Hello World" program for our TV.
So create a file like this:
hellow.c
#include <stdio.h>
int main(void) {
printf("Hello world\n");
return (0);
}
We now, need to determine what compiler flags to use with our cross-compiler. There are several dozens of compiler options for the CodeSourcery compiler, but we are only interested in the following.
Here are the most important CodeSorcery ARM Cross Compiler options:
-march= Specify the name of the target architecture -mcpu= Specify the name of the target CPU -mfpu= Specify the name of the target FPU hardware/format -marm Generate code in 32 bit ARM state. -mthumb Generate code for Thumb state -mlittle-endian Assume target CPU is configured as little endian -mthumb-interwork Support calls between Thumb and ARM instruction -mglibc Use GNU C library -muclibc Use uClibc C library -static Compile and include all static libraries
Here are the choices for the above options:
Known ARM ABIs (for use with the -mabi= option): aapcs aapcs-linux apcs-gnu atpcs iwmmxt Known ARM architectures (for use with the -march= option): armv2 armv2a armv3 armv3m armv4 armv4t armv5 armv5e armv5t armv5te armv6 armv6-m armv6j armv6k armv6s-m armv6t2 armv6z armv6zk armv7 armv7-a armv7-m armv7-r armv7e-m ep9312 iwmmxt iwmmxt2 native Known ARM CPUs (for use with the -mcpu= and -mtune= options): arm1020e arm1020t arm1022e arm1026ej-s arm10e arm10tdmi arm1136j-s arm1136jf-s arm1156t2-s arm1156t2f-s arm1176jz-s arm1176jzf-s arm2 arm250 arm3 arm6 arm60 arm600 arm610 arm620 arm7 arm70 arm700 arm700i arm710 arm7100 arm710c arm710t arm720 arm720t arm740t arm7500 arm7500fe arm7d arm7di arm7dm arm7dmi arm7m arm7tdmi arm7tdmi-s arm8 arm810 arm9 arm920 arm920t arm922t arm926ej-s arm940t arm946e-s arm966e-s arm968e-s arm9e arm9tdmi cortex-a15 cortex-a5 cortex-a7 cortex-a8 cortex-a9 cortex-m0 cortex-m1 cortex-m3 cortex-m4 cortex-r4 cortex-r4f cortex-r5 ep9312 fa526 fa606te fa626 fa626te fa726te fmp626 generic-armv7-a iwmmxt iwmmxt2 mpcore mpcorenovfp native strongarm strongarm110 strongarm1100 strongarm1110 xscale Known ARM FPUs (for use with the -mfpu= option): fpa fpe2 fpe3 fpv4-sp-d16 maverick neon neon-fp16 neon-vfpv4 vfp vfp3 vfpv3 vfpv3-d16 vfpv3-d16-fp16 vfpv3-fp16 vfpv3xd vfpv3xd-fp16 vfpv4 vfpv4-d16
ARMed with our previous knowledge from part (B) we can try the following:
$ arm-none-linux-gnueabi-gcc.exe -march=armv7-a -mcpu=cortex-a9 -marm -mlittle-endian -mglibc -static hellow.c -o hellows $ arm-none-linux-gnueabi-gcc.exe -march=armv7-a -mcpu=cortex-a9 -marm -mlittle-endian -mglibc hellow.c -o hellowd
Great! It seem to work. But as you can see, a statically compiled binary is about 100x bigger than a dynamically compiled one of size ~10K. But sometimes we need a static binary as it can help overcome crippled, buggy or platform specific system libraries.
But let's check if we got what we expected:
$ file hellows hellows: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.16, not stripped $ file hellowd hellowd: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, not stripped $ arm-none-linux-gnueabi-objdump.exe -x hellows |less ...
Looks good, let's try to run it. ( /dtv/usb/sda1 )
ftp <tv_ip> ftp> put hellows /tmp/bin/hellows ftp> put hellowd /tmp/bin/hellowd ftp> quit nc <tv_ip> 23 shell> chmod 777 /tmp/bin/hellow* shell> hellows Hello world shell> hellowd Hello world
Excellent! We are now ready to try a more advanced example where we will make use of some platform specific system libraries to make a kernel module.
(D) Installing the Samsung Kernel Sources
Download the sources relevant to your TV set from the Samsung Open Source repository.
In our case (UE40ES5700) it would be UExxES6xxx.zip. For other ES models, it would be UNxxES8xxx.zip.
But these two are exactly the same, except the following:
1. E8 and E6 have slightly different VDLinux kernels 2. There is an additional OR1200.ZIP for the E8
The only files (~150) which are different are listed HERE.
After download, do test the downloaded zip archive with:
$ unzip -t UExxES6xxx.zip
If you are curious to see detailed info of what's inside the ZIP archive before unzipping, do this:
$ unzip -Z UExxES6xxx.zip ...
Extract the ZIP file to a directory where you would like to keep your sources.
$ cd /zarm/src/ $ unzip UExxES6xxx.zip -d UExxES6xxx
This will unzip the following:
alsa-lib-1.0.23.tgz Advanced Linux Sound Architecture (audio and MIDI)
ATK.tgz Accessibility Toolkit (screen User Interface)
binutils-2010q1.tgz A collection of binary tools (ld, as, nm, objdump etc.)
BROADCOM-bthid.tgz Broadcom Bluetooth HID drivers (keyboards, mice, game controllers)
BROADCOM-btusb.tgz Broadcom Bluetooth USB drivers (keyboards, mice, game controllers)
busybox-1.18.1.tgz Busybox combines many common UNIX utilities into a single executable
Cairo.tgz A 2D graphics library (X Window, quartz, win32, PDF, PS, SVG file output)
FFMPEG.tar.gz A cross-platform solution to record, convert and stream audio and video
glibc-2.11-2010q1.tgz The GNU C Library
Glibmm.tgz A C++ interface for the popular cross-platform library Glib
gnutls-2.6.4.tar.gz GNU Transport Layer Security Library (SSL, TLS and DTLS protocols)
iptables-1.4.10.tgz Linux kernel firewall
libgcrypt-1.4.5.tar.gz A general purpose cryptographic library
libgpg-error-1.7.tar.gz A library that defines common error values for all GnuPG components
LIBGPHOTO2.tar The core library designed to allow access to digital camera by external programs
LibMMS_0.6.2.tgz A library for parsing mms:// and mmsh:// type network streams
libsoup.20120109.tgz An HTTP client/server library for GNOME
libtasn1-2.5.tar.gz The ASN.1 library used by GnuTLS
LIBUSB.tar A C library that gives applications easy access to USB devices
Pango.tgz A library for layout and rendering of multi-language text
RALINK_RTNET5572STA_V_2_5_0_1.tgz Ralink RTnet RT5572 (Wifi USB dongle drivers)
RALINK_RTUTIL5572STA_V_2_5_0_1.tgz Ralink RTnet RT5572 (Wifi USB dongle utilities)
readme.zip HOW_TO_BUILD_X9X10.txt
SDL.tar.gz Simple DirectMedia Layer (a multimedia library written in C)
uvc.tar.gz USB Video Class (streaming webcams, digital cameras etc)
v4l2.tar.gz Video4Linux-2 (a video capture API for Linux)
VDLinux_2.6.35.11.tgz Kernel sources (VDLinux, Tuxera NTFS, RFS, LinuStoreIII, FSR)
webkit-gtk.20120109.tgz WebKit is an open source web browser engine (Safari, Chrome)
WIRELESSTOOLS_29.tgz Wireless Tools (iwconfig, iwlist, etc)
xfsprogs-3.1.5.tgz A set of command-line tools to manage XFS filesystems
Which of these do you really need? Now that is the million dollar question.
The simple and stupid answer is, that it depends on what you want to do.
a) If you need to compile some simple C-code using standard clibs and linux system calls, you probably don't need any. (E.g. helloworld.c) b) If you need to compile your own Kernel module (hellok.ko) you probably need: VDLinux_2.6.35.11.tgz c) If you want to compile your own Kernel image (uImage) you probably only need: VDLinux_2.6.35.11.tgz d) If you want to compile your own library object (somelib.so) you probably only need: VDLinux_2.6.35.11.tgz ++++ e) If you need to compile your own device driver you probably need: <the device driver sources> VDLinux_2.6.35.11.tgz ? glibc-2.11-2010q1.tgz ? Glibmm.tgz f) If you want to compile your own TV DSP (exeDSP, micom etc) you're screwed (by Samsung) since there are no publicly available sources for that.
The better answer is, that it depends on how your kernel image has been setup
and how you intend setup your compilation environment. We really don't want to
have to setup and compile all sources from scratch, just to make a simple kernel
module. The way to do that is by telling your cross-compiler where to find the
kernel header files that it has to use the same configuration as that used to
compile your kernel.
So for further details, we look at HOW_TO_BUILD_X9X10.txt inside the readme.zip. This file explains, in a very screwy way, how to build each of the items included on the UExxES6xxx.zip file. In fact you should not rely blindly on this info.
For example:
... [ Building linux kernel ] * Source code name : VDLinux_2.6.35.11.tgz * Unpack the kernel tarball and cd into it. * Run "cd VDLinux_2.6.35.11/linux-2.6.35.11/". * Run "cp -ar arch/arm/configs/X10P_defconfig_release .config". * Run "make oldconfig". * Run "make uImage". [ Building busybox ] * Source code name : busybox-1.18.1.tgz * Unpack the busybox tarball and cd into it. * Run "make clean". * Run "cp -ar configs/busybox_config .config". * Run "make ARCH=arm CROSS_COMPILE=arm-v7a8v2r2-linux-gnueabi- oldconfig". * Run "make ARCH=arm CROSS_COMPILE=arm-v7a8v2r2-linux-gnueabi- CONFIG_PREFIX=../temp_rootfs install". * Run "cd ../temp_rootfs/bin". ...
Clearly, for general purpose use we need at least the VDLinux sources installed.
So we extract this in the same directory with:
$ (tar -zxvf VDLinux_2.6.35.11.tgz >vdlinux_tar.log) 2>&1
Here we have redirected the output to a log file for reference, while any
errors will be shown on screen. This will create VDLinux_2.6.35.11
with the following sub-directories:
linux-2.6.35.11
TUXERA_NTFS
RFS_3.0.0_b043-LinuStoreIII_1.2.0_b039-FSR_1.2.1p1_b139_RTM
The last two directories are soft linked into the ./tntfs and ./rfs sub-directories of ./linux-2.6.35.11. However, these links are not well re-made (especially after decompression). So if you think you'll need them, it is probably better to just copy them into there...
< more details needed >
WIP!!
placeholder
(E) Building the Kernel using Cygwin
If you're not using Cygwin, you can skip directly to: Cross-compiling_(ES_series)#Compiling_the_Kernel
!! STOP !! First read these REQUIREMENTS!!
Before proceeding, read the following very carefully! I mean it!
We strongly discourage the use of Mentor Graphics (Aka. CodeSourcery)
CodeBench Lite ARM cross-compilers, for anything other than compiling
simple freestanding programs. If you intend to write a simple kernel module or
kernel device driver, or any other less trivial development, you "must" use a
native linux environment. Although Cygwin provides for an almost-linux
environment, the quirks introduced by mixing the various Windows-Cygwin-Linux
tools, creates a huge headache, that most people should avoid.
Below we will show you that you can indeed compile a Linux Kernel using Cygwin
and some of the tools mentioned above. But the road to get there is a horrible
waste of time and energy, trying to patch and resolve problems that should not
be there in the first palce. Do not bother to ask in the forums for support
using Cygwin or CodeBench + Windows combination. You have been warned!
Instead follow the instructions: Cross-compiling VM-style (ES)
REQUIREMENTS!!
If you intend to use Cygwin with the Sourcery CodeBench (a pre-compiled windows executable) ARM cross-compiler, do not believe for one second, that you will be successful compiling anything other than HelloWorld.c for your Linux based ARM (e.g. Android phone, SmartTV etc.) device, unless you follow these requirements.
First let me tell you what the main problems are with using windows.
Standard Windows (XP, Vista, Windows-7):
- Uses ACL to set file permissions and ownership (Not Linux compatible.)
- Has it's own way of creating symbolic links (Not Linux compatible.)
- Uses a non case-sensitive default for its fixed NTFS drives.
(Often and silently break archives originally compressed under Linux.)
- Uses the Win32 (non POSIX) standard for file paths (Not Linux compatible.)
- Uses the 2 characters Carriage-Return and New-Line ("\r" & "\n") for
End-of-Line (EOL) representation, contrary to POSIX, which uses only NL.
If you're going to compile a Linux Kernel on Cygwin, you have to first resolve
the above Windows problems. Second, you'll need to be aware of other Cygwin quirks,
that could affect your success. Here is how.
(a) "Disable" the native Windows use of Access Control Lists (ACL).
This is probably not completely correct/possible, but by making yourself
a permanent Administrator, it will have the same effect. If you have ever
owned a Vista machine, you have probably already done this, as it created
huge problems in the past. Google it! Apparently Cygwin has a bug that
doesn't take into account default ACL settings and "umask" gives the wrong
permissions. So you have to change permissions and ownerships manually
with chmod/chown. There is one other possibility (not verified). You can
try to mount your Cygwin partitions in /etc/fstab with the "noacl" flag:
none /cygdrive cygdrive binary,noacl,posix=1,user 0 0
But see below first. If you do insist on messing with Windows ACL's check
out commands like: "getfacl, setfacl, cacls" and to recursively find and
set all Users files ACLs to "rwx", use:
"find /cygdrive/c/Users -exec setfacl -r -m default:other:rwx {} \;"
Or read all about it here: http://tiny.cc/pr9yqw & http://tiny.cc/rwczqw
(b) Enable NTFS case-sensitivity by editing the Windows registry.
We are constantly warned by MS for unexpected results of doing this, and
that it could have grave consequences for your system. That is complete
BS, as the only unexpected consequence is that you start wondering why you
didn't re-enable this idiotic Windows "feature" ages ago. I certainly
questioned what the fcuk was going on every time I extracted some Linux
tarball, and kept getting dozens of files having the "same" name and given
the "choice" of renaming them! (WTF! They already had different names, as
Ab.txt =/= aB.txt !) If you choose not to "rename" them, only the last one
extracted will survive! You will never know! Do this, in your Cygwin Bash
shell:
$ regedit.exe &
--> navigate to: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\kernel]
--> add the following item: obcaseinsensitive REG_DWORD 0
Now reboot and and let your PC "settle", by running some of your
favorite apps. No problems? Good, reboot again and you're done.
NOTE: Cygwin online documentation claim two things that are not true:
(1) that you can temporarily remount a disk as "posix=1" to enable
case sensitivity. This doesn't work, and even if it did, you
would immediately loose this, after closing last Cygwin shell.
It only works if NTFS case-sensitivity is already enabled, but
have been disabled when default mounted in /etc/fstab.
(2) "mount" only shows disk as "posix=0" (case insensitive),
and never as "posix=1" (case sensitive), as claimed by online
documentation.
So until you have implemented a case sensitive system, avoid developing
Linux systems using tarballs extracted under windows.
(c) Windows have it's own screwed up way of doing symbolic links, that are
not compatible with linux symbolic links. A Win32 symlink is just a file
with a special tag followed by the file inode address AFAIK. But Cygwin
can't handle this very well, so that if you symlink a regular file, your
Cygwin application will fail, reading the content, instead of following
the link. But if you symlink a directory, it works. This is probably
a Cygwin bug. But it doesn't matter, because if you use original symlinks
(from a Linux tarball, for example) you will get fcuked by the Win32 API
anyway, regarding both file permissions and ownerships.
Instead, just find and delete all symlinks, and try to recreate them
within Cygwin. But to be sure, just copy the files/directories instead.
(This is specially important if you try to compile linux kernels with
thousands of files, some which may be symlinked.)
To find all symlinks use: "find ./ -type l"
So NEVER extract any source code tarballs using Windows utilities such
as Winzip, 7zip etc, as they will fail to re-create proper hard links.
(d) One major obstacle, when trying to use a (native windows) pre-compiled
cross-compiler with Cygwin, is that they use Win32 file paths. These are
not normally compatible with Linux based tools like binutils and make,
which require POSIX paths. Cygwin has it's own internal mechanism for
dealing with Win32 paths, through /bin/cygpath.exe. This program can
convert back and forth. But the problem occurs when POSIX programs like
"make" is using windows cross-compilers (e.g. Sourcery CodeBench Lite)
which return results with Win32 paths. According to their documentation
this can be handled by defining the shell variable "CYGPATH=cygpath".
However, this is not working as described and expected, as easily
demonstrated with:
$ arm-none-linux-gnueabi-gcc.exe --print-sysroot
d:\zarm\csbench\bin\../arm-none-linux-gnueabi/libc
So you might think that adding the "--sysroot" flag might help. Well it
does somewhat: but not completely:
$ arm-none-linux-gnueabi-gcc.exe --sysroot=/cygdrive/d/zarm/ -print-sysroot
/cygdrive/d/zarm/
becuase if you do the same with "-print-file-name" which is used in the
top Makefile, you still get a Win32 path!
$ arm-none-linux-gnueabi-gcc.exe -print-file-name=include
d:/zarm/csbench/bin/../lib/gcc/arm-none-linux-gnueabi/4.7.2/include
But apparently this is not all. It seem that CodeBench is also "leaking"
out other internal variables that are poisoning the environment with Win32
pathnames that have been compiled in. For example, if you do the following:
arm-none-linux-gnueabi-gcc.exe -v --help >/dev/null
You'll get things like:
COLLECT_GCC=D:\zarm\csbench\bin\arm-none-linux-gnueabi-gcc.exe
COLLECT_LTO_WRAPPER=d:/zarm/csbench/bin/../libexec/gcc/arm-none-linux-gnueabi/4.7.2/lto-wrapper.exe
COLLECT_GCC_OPTIONS
COMPILER_PATH
LIBRARY_PATH
and whole bunch more. But these are not a problem...until they start popping
up in your assembly code. The most interesting of these are the "-iprefix"
and "-isysroot" which seem to propagate into assembler without anyone ever
entering these. For example, in the file ./kernel/bounds.s we find:
@ -iprefix d:\zarm\csbench\bin\../lib/gcc/arm-none-linux-gnueabi/4.7.2/
@ -isysroot /cygdrive/d/zarm/csbench/arm-none-linux-gnueabi/libc
...
@ -isystem /cygdrive/d/zarm/csbench/lib/gcc/arm-none-linux-gnueabi/4.7.2/include
@ -include include/generated/autoconf.h -MMD kernel/.bounds.s.d
The only soultion is to manually specify new paths for both -sysroot and -iprefix.
If you use both this option and the -isysroot option, then the -sysroot option
will apply to libraries, but the -isysroot option will apply to header files.
We do this in the Makefile.
(e) Then we have the issue with differing End-of-Line (EOL) character(s) on POSIX
based systems like Cygwin (and Linux) and that used by windows. Where Win32 uses
the 2 character combination of a Carriage-Return and a New-Line ("\r" & "\n")
for EOL representation, POSIX ony use NL ("\n"). This has serious consequences
for the in-between compilation, where GCC is automatically creating link scripts
to be used later in the linking process. Thus the linking scripts need to be
converted. This is done by the "unix2dos" and "dos2unix" utilities. But we
have to patch the Makefiles to use these.
(f) The final Cygwin quirk relates to the automatic recognition of .exe files.
As you know, Cygwin treats .exe files on equal footing as non-.exe files,
almost! Basically if you have a script called "myprog" and an executable
called "myprog.exe", then "myprog.exe" takes precedence when typing "myprog".
http://cygwin.com/cygwin-ug-net/cygwin-ug-net.html
http://seit.unsw.adfa.edu.au/staff/sites/hrp/webDesignHelp/cygwin-ug-net-nochunks.html#USING-CYGWINENV
"Executable program filenames end with .exe but the .exe need not be included in
the command, so that traditional UNIX names can be used. However, for programs
that end in ".bat" and ".com", you cannot omit the extension."
As a side effect, the ls filename gives information about filename.exe if
filename.exe exists and filename does not. In the same situation the function
call stat("filename",..) gives information about filename.exe. The two files can
be distinguished by examining their inodes, as demonstrated below.
If a shell script myprog and a program myprog.exe coexist in a directory, the
program has precedence and is selected for execution of myprog.
The gcc compiler produces an executable named filename.exe when asked to produce
filename. This allows many makefiles written for UNIX systems to work well under
Cygwin.
Unfortunately, the install and strip commands do distinguish between filename
and filename.exe. They fail when working on a non-existing filename even if
filename.exe exists, thus breaking some makefiles. This problem can be solved by
writing install and strip shell scripts to provide the extension ".exe" when
needed.
Did you read and understand all that? Good, then you are ready to patch the Supplied Samsung kernel.
I have actually patched it for you, but you should read through to see what was done. (Download link at the end.)
Setting up your Cygwin environment
The BASH Shell environment
We need to make sure our Cygwin Bash shell environment is as close as possible to what you'd find on a typical Linux distribution. The way to find out what those settings are is to issue the following command, which will display the enabled and disabled Bash shell environment settings. contained in the $BASHOPTS and $SHELLOPTS environment variables.
set -o && echo -e "\n" && shopt -p
The $BASHOPTS and $SHELLOPTS environment variables only show the enabled ones.
$ echo "SHELLOPTS=$SHELLOPTS"
SHELLOPTS=braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor
$ echo "BASHOPTS=$BASHOPTS"
BASHOPTS=checkwinsize:cmdhist:expand_aliases:extglob:extquote:force_fignore:histappend:
interactive_comments:login_shell:progcomp:promptvars:sourcepath
Make them match these, and you should be ok.
The Cygwin Specific variables
Next you should set these two variables.
export CYGWIN=nodosfilewarning export CYGPATH=cygpath
The first one disables the Warning message that appears when Cygwin detects files using DOS EOL. While the second sets the path used for CodeBench to find the cygpath.exe path conversion program.
The Compiler Specific variables
Then to avoid having to type very long command lines for just doing a "make", we define the following:
export ARCH=arm
export CROSS_COMPILE=arm-none-linux-gnueabi-
export CONFIG_CROSS_COMPILE=${CROSS_COMPILE}
export KCFLAGS="-imultilib ./ -mno-unaligned-access -mlittle-endian \
-marm -mapcs -mabi=aapcs-linux -mno-thumb-interwork \
-march=armv7-a -mfloat-abi=softfp -mtune=cortex-a8 -mfpu=vfp3"
The setup script
All of the above (and more) is put in our setup script.
sexport.sh:
#! /bin/sh
# sexport.sh - Exports variables used by the CodeSorcery ARM cross-compiler
#==============================================================================
# NOTE: Export doesn't work when used in shell script. It can only be "sourced"
# So you need to run this with: ". ./sexport.sh"
#------------------------------------------------------------------------------
checkwindrives () {
MYOS=`uname -o`
if [[ $MYOS == "Cygwin" ]]; then
TEST=`mount |grep -i "posix=0"`
if [ $? != 1 ]; then
echo -e "One or more of your harddrives are mounted as NOT case-sensitive!\n";
echo -e "$TEST\n";
echo "A case sensitive NTFS drive may be necessary for compiling Linux Kernels on Cygwin.";
echo "You can fix this by enabling case sensitivity in the windows registry and rebooting:";
echo " HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\ DWORD:obcaseinsensitive 0";
echo "You can also try to remounting cygdrives in 'posix=1' mode by editing /etc/fstab to:";
echo " 'none /cygdrive cygdrive binary,posix=1,user 0 0'"
echo "or try directly on command line (after reboot) with:";
echo " 'umount /cygdrive/d'";
echo " 'mount -f -o binary,posix=1 d: /cygdrive/d'";
echo "WARNING: Doing this may have unknown consequences for other native windows programs.";
echo " See: http://tiny.cc/4s0uqw and http://tiny.cc/ik1uqw and http://tiny.cc/4oavqw ";
fi
fi
}
if [ $0 != "bash" ]; then
echo "This export script need to be sourced!";
echo "That is, you have to run it with: . ./sexport.sh"
exit 1;
else
export ARCH=arm
export CROSS_COMPILE=arm-none-linux-gnueabi-
export CONFIG_CROSS_COMPILE=${CROSS_COMPILE}
export PATH=${PATH}:/cygdrive/d/zarm/devbin/bin:.
export SAMYGO=arm-samygo-linux-gnueabi-
export SYSROOT=/cygdrive/d/zarm/csbench/arm-none-linux-gnueabi/libc
export KCFLAGS="-imultilib ./ -mno-unaligned-access -mlittle-endian \
-marm -mapcs -mabi=aapcs-linux -mno-thumb-interwork \
-march=armv7-a -mfloat-abi=softfp -mtune=cortex-a8 -mfpu=vfp3"
MODHOME=/cygdrive/d/zarm/myarm/kernel_modules
export INSTALL_PATH=${MODHOME}/kimagemap
export MODLIB=${MODHOME}/kmods
export INSTALL_MOD_PATH=/
shopt -s checkwinsize
shopt -s extglob
shopt -u hostcomplete
echo -e "\nExported:\n "
echo " ARCH=$ARCH"
echo " CROSS_COMPILE=$CROSS_COMPILE"
echo " CONFIG_CROSS_COMPILE=$CONFIG_CROSS_COMPILE"
echo " SAMYGO=$SAMYGO"
echo;
echo " GCC_EXEC_PREFIX=${GCC_EXEC_PREFIX}";
echo " SYSROOT=${SYSROOT}";
echo " LD_RUN_PATH=${LD_RUN_PATH}";
echo;
# echo " KCPPFLAGS=$KCPPFLAGS"
echo " KAFLAGS=$KAFLAGS"
echo " KCFLAGS=$KCFLAGS"
echo;
echo " INSTALL_PATH=${INSTALL_PATH}"
echo " INSTALL_MOD_PATH=${INSTALL_MOD_PATH}"
echo " MODLIB=${MODLIB}"
echo;
echo "You can use: 'export -n <VAR>' OR: 'unset <VAR>' to remove property.";
echo "Also confirm that 'alias'es are not masking any important commands. Use 'unalias' to fix.";
echo;
# echo " SHELL=${SHELL}";
echo " CYGPATH=${CYGPATH}";
echo " CYGWIN=${CYGWIN}";
echo;
echo "Enabled BASH(OPTS) environment variables:";
echo -e "\n ${BASHOPTS}\n" |sed -r 's/\:/\n /g'
echo "Enabled SHELL(OPTS) environment variables:";
echo -e "\n ${SHELLOPTS}\n" |sed -r 's/\:/\n /g'
echo "Use 'set +o <shell_opt>' and 'shopt -u <bash_opt>' to disable.";
echo "Use 'set -o <shell_opt>' and 'shopt -s <bash_opt>' to enable.";
echo;
# Check locally mounted drives for case sensitivity (Cygwin only):
checkwindrives
fi
Obviously you have to edit the paths to what you have...
Patching the Kernel for Cygwin use
We will not show the boring details for this patching here.
Please have look in our forum HERE for all the gory details.
Since there is no available info how to patch VDLinux (based on 2.6.35.13) for Cygwin compilation, the patches used to get this working was handpicked and applied by hand from the sources below. It is not clear as to what patches are actually needed, and others may overlap in their functionality.
The patches are based on the following:
[1] https://sourcery.mentor.com/GNUToolchain/kbentry21 [2] http://communities.mentor.com/community/cs/archives/arm-gnu/msg01267.html [3] http://communities.mentor.com/community/cs/archives/arm-gnu/msg01799.html [4] HOWTO: Linux kernel compilation for ARM using cygwin (01 Aug 2007) [5] HOWTO: cross-compile the Linux kernel on Cygwin (11 Jun 2012) [6] Cygwin User guide [7] Google: "cygwin kernel compile error site:communities.mentor.com/community/cs/archives/arm-gnu/" [8] Patch for Toolchain 4.6.3-cygwin (perhaps not needed)
Download and install the patched VDLinux
NOTE: This is not the same as the "Patched Firmware" in the top E-Series Wiki page.
This patch is only for use with Cygwin + CodeBench Lite (windows) cross compiler!!
blah blah
(F) Compiling the Kernel
Before you build your own kernel module, it will help to first make sure you can build a kernel image. Although this is not necessary, it will help with setting up some required automatically generated scripts. These scripts generate some special header files that contain the Linux kernel versions and "vermagic" that might be needed for your kernel module to work.
Checking your Kernel version
The special files with version information are:
./include/linux/version.h: #define LINUX_VERSION_CODE 132643 #define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c)) ./include/generated/utsrelease.h: #define UTS_RELEASE "2.6.35.13" ./include/linux/vdlp_version.h: (is symlinked to: vdlp_version_X10P.h ) #define DTV_KERNEL_VERSION "0062, release" #define DTV_LAST_PATCH "0713, DTV, X10P, release, AUS_BRANCH"
Find a kernel module from either the TV itself or from downloading and extracting the firmware. Then find your current Kernel version by:
strings <some-module>.ko |grep "vermagic"
For example:
$ strings drivers/hid/hid.ko |grep "vermagic" vermagic=0062, release SMP preempt mod_unload ARMv7
Then edit the file above by changing the DTV_KERNEL_VERSION value to what you found.
(G) Compiling a Kernel module
Introduction
Linux is a monolithic kernel, where all of the code and data that makes up the
image are linked into one binary and loaded into memory.
"A very useful part of the Linux kernel architecture is the support for
loadable kernel modules. These modules allow the otherwise monolithic kernel
to be split up into smaller components that can later be loaded as required,
allowing the kernel to ship with support for a wide range features but only
load those that are needed.
Kernel modules also ease the development of new features such as file systems
or device drivers, as a new experimental modules can be quickly built, loaded
into a basic kernel, exercised, and then unloaded. This is much faster then
the build, flash, restart process that would otherwise be required."
The main reason for this, is that you don't have to re-compile the entire
kernel from the full kernel source tree. It should be enough to just have
the kernel header files, in order to build a module.
Internally a module is a standard ELF executable file with a .ko extension and
a few special sections such as .modinfo for the module metadata and .init.text
for the module initialization code. (This is normally done by linking
yourprogram.o with vermagic.o, which is transparent to the developer.)
A nice thing about modules being ELF files is that they can be generated
and inspected by standard tools.
In the current 2.6 series, the ARM kernel is laid out as follows:
Start End Contents ----------------------------------------------------------------------------- 0xFF000000 0xFFFFFFFF Vector page, DMA region, and others VMALLOC_END 0xFEFFFFFF free VMALLOC_START VMALLOC_END vmalloc() / ioremap() space PAGE_OFFSET high_memory The Linux kernel TASK_SIZE PAGE_OFFSET-1 Kernel module space (16 MB) 0x00001000 TASK_SIZE-1 User space (~3 GB) 0x00000000 0x00001000 Vector page / Null pointer trap ----------------------------------------------------------------------------- PAGE_OFFSET = 0xC0000000 TASK_SIZE = 0xBF000000
Note that these are virtual addresses, which are different than the physical address space of the board.
The HelloWorld-1 kernel module
We will attempt to build two kernel modules. One very basic in "Hello World"
style to see that it compiles and works,
and another only slighlty more complicated to check if kernel debugfs works.
Any module which want to send info to kernel messages ( /proc/kmsg ) have
to include the kernel.h header file.
What is actually shown depend on the current kernel debug level as set in /proc/printk...
./include/linux/kernel.h:
... #define KERN_EMERG "<0>" /* system is unusable */ #define KERN_ALERT "<1>" /* action must be taken immediately */ #define KERN_CRIT "<2>" /* critical conditions */ #define KERN_ERR "<3>" /* error conditions */ #define KERN_WARNING "<4>" /* warning conditions */ #define KERN_NOTICE "<5>" /* normal but significant condition */ #define KERN_INFO "<6>" /* informational */ #define KERN_DEBUG "<7>" /* debug-level messages */ ...
You can see your current kernel debug level with either:
shell> sysctl -a ... kernel.printk= ... OR shell> cat /proc/loglevel ??? <== check!! shell> cat /proc/printk kernel.printk = 4 4 1 7
You can read more about these here:
modinfo -p ${modulename}
echo -n ${value} > /sys/module/${modulename}/parameters/${parm}
Now, to the actual kernel module code.
hellok1.c:
#include <linux/module.h> /* Needed by all modules */
#include <linux/kernel.h> /* Needed for KERN_ALERT */
MODULE_LICENSE("GPL");
MODULE_AUTHOR("E:V:A 2013");
MODULE_DESCRIPTION("Demo kernel module for MST-X10P (ARM Cortex A9)");
int init_module(void) {
printk(KERN_ALERT "E:V:A is in the Kernel!\n");
return 0;
}
void cleanup_module(void) {
printk(KERN_ALERT "Goodbye TV Kernel!\n");
}
Now that was the trivial part!
The Makefile
The most difficult part of compiling a kernel module, is setting up the Makefile that contain all the compilation instructions, locations and parameters needed for your cross-compiler. It is good to be familiar with the "Makefile" language, as it is tab and space sensitive, and have many other pitfalls, that can easily be overseen. The Makefile is also closely connected with how your kernel have been compiled, so if you're missing kernel support for your modules features (e.g. debugfs), you will not get anything... (It should be noted that Makefile may also contain the functionality of Kbuild, which is very similar, but whose structure is even more simple. We will not cover the details of this here.)
Makefile:
ifneq ($(KERNELRELEASE),) obj-m += hellok1.o else KERNELDIR := /your/path/to/linux-2.6.35.11/ all: $(MAKE) -C $(KERNELDIR) M=$(PWD) modules clean: rm -fr ./.tmp_versions Module.symvers modules.order ./.*.cmd ./*.mod.* ls -al endif
NOTE: The code above contains hidden tabs, so make sure they're still there if you decide to copy & paste the code from above!
Compiling the module
Go to your directory containing your Makefile and module programs.
Minimally containing:
Makefile hellok1.c hellok2.c
Then just type "make". (Make sure you have correctly setup your BASH environment variables.)
Inserting the Module
Copy the module to your TV in the same way we did in the beginning.
For our TV set we have in dmesg:
...
SAMSUNG VDLP Kernel
Version : 0064, release
Applied Last Patch Number : 0716, DTV, X10P, release, DEU_BRANCH
...
So if we try to insert a kernel module with the wrong vermagic, we get an error message in "dmesg":
shell> /tmp/bin/busybox insmod /dtv/usb/sda1/hellok1.ko shell> dmesg ... hellok1: version magic '0062, release SMP preempt mod_unload ARMv7 ' should be '0064, release SMP preempt mod_unload ARMv7 ' ...
We edit the version.h file (mentioned above) and recompile our module. This time "dmesg" greets us with:
... E:V:A is in the Kernel! hellok1 mod ld ...
You can then see your module in the FS:
shell> ls -l /sys/module/hellok1 drwxr-xr-x 2 root 0 0 Jan 1 00:41 holders -r--r--r-- 1 root 0 4096 Jan 1 00:41 initstate -r--r--r-- 1 root 0 4096 Jan 1 00:41 refcnt -r--r--r-- 1 root 0 4096 Jan 1 00:41 srcversion
shell> cat /sys/module/hellok1/initstate live
To remove your module from the kernel use:
shell> rmmod hellok1
In "dmesg":
... Goodbye TV Kernel! hellok1 mod uld ...
If you try to load the hellok2.ko module on a linux kernel configured without DEBUGFS, you'll get this message (dmesg):
... hellok2: Unknown symbol debugfs_remove_recursive (err 0) hellok2: Unknown symbol debugfs_create_file (err 0) hellok2: Unknown symbol debugfs_create_dir (err 0) ...
The HelloWorld-2 kernel module
Now that we have gained some intellectual kernel meat, we can design a slightly more useful kernel module. One that actually does something.
The following code is an example kernel module that shows the life cycle and provides a function to user space over debugfs. It creates a new file in /sys/kernel/debug/hello/ping that can be opened and read from user space. Reading this file gives a short message that includes the number of times the file has been opened. All original comments have been removed, but can be found in [3].
hellok2.c:
#include <linux/init.h>
#include <linux/module.h>
#include <linux/debugfs.h>
#include <linux/seq_file.h>
MODULE_LICENSE("Dual BSD/GPL");
static struct dentry *root_dir;
static int calls;
static int hello_print(struct seq_file *s, void *p) {
seq_printf(s, "Called %d times\n", ++calls);
return 0;
}
static int hello_open(struct inode *inode, struct file *file) {
return single_open(file, hello_print, inode->i_private);
}
static const struct file_operations hello_fops = {
.open = hello_open,
.write = NULL,
.read = seq_read,
.llseek = seq_lseek,
.owner = THIS_MODULE,
};
static int hello_init(void) {
printk(KERN_ALERT "Hello world\n");
root_dir = debugfs_create_dir("hello", NULL);
debugfs_create_file("ping", 0444, root_dir, NULL, &hello_fops);
return 0;
}
static void hello_exit(void) {
debugfs_remove_recursive(root_dir);
printk(KERN_ALERT "Goodbye Dear Module\n");
}
module_init(hello_init);
module_exit(hello_exit);
The Makefile for this is your homework, but a good hint is that the tricky part this time, is not the Makefile, but enabling DEBUGFS in your kernel, if not already enabled...
Enabling DEBUGFS in Kernel
Enter the kernel configuration menu:
make menuconfig
and enable the following:
(a) Go to the "Kernel Features" section.
Enable ARM EABI:
CONFIG_AEABI=y
(b) Go to the "Kernel hacking" section.
1. Enable the Debug Filesystem:
CONFIG_DEBUG_FS=y
2. Enable Kernel debugging. The screen will change and a
number of new, debug-related options will come visible.
CONFIG_DEBUG_KERNEL=y
3. Enable Compile the kernel with debug info. This turns on the
standard -g option to the compiler and linker and expands the
vmlinux image to include all the symbolic, line-level debugging
information needed later.
CONFIG_DEBUG_INFO=y
Note: (2) and (3) are not recommended for a production kernels.
Save and exit. Then recompile your kernel.
WIP!! ....
References
CodeSourcery Application Notes:
[1] "Flying Introduction to Linux Kernel Development" (AN001)
[2] "Using Sourcery CodeBench to Debug the Linux Kernel" (AN002)
[3] "Using Sourcery CodeBench to Develop and Debug a Linux Kernel Module" (AN003)
[4] "GNU make manual"
[5] "Kernel.org Makefile manual"
[6] "seq_file HowTo"
Common Make Variables
Some commonly-used predefined "make" variables:
AR default 'r' Archive-maintaining program AS default 's' Program for compiling assembly files CC default 'c' Program for compiling C programs CXX default '++' Program for compiling C++ programs CPP default '(CC) -E' Program for running the C preprocessor FC default '77' Program for compiling Fortran and Ratfor programs M2C default '2c' Program to use to compile Modula-2 source code PC default 'c' Program for compiling Pascal programs CO default 'o' Program for extracting a file from RCS GET default 'et' Program for extracting a file from SCCS LEX default 'ex' Program to use to turn Lex grammars into source code YACC default 'acc' Program to use to turn Yacc grammars into source code LINT default 'int' Program to use to run lint on source code MAKEINFO default 'akeinfo' Program to convert a Texinfo source file to Info file TEX default 'ex' Program to make TeX dvi files from TeX source TEXI2DVI default 'exi2dvi' Program to make TeX dvi files from Texinfo source WEAVE default 'eave' Program to translate Web into TeX CWEAVE default 'weave' Program to translate C Web into TeX TANGLE default 'angle' Program to translate Web into Pascal CTANGLE default 'tangle' Program to translate C Web into C RM default 'm -f' Command to remove a file # ------------------------------------------------------------------ ARFLAGS Extra flags to the archive-maintaining program; default 'rv'. ASFLAGS Extra flags to the assembler when invoked on a '.s' or '.S' file. CFLAGS Extra flags to the C compiler. CXXFLAGS Extra flags to the C++ compiler. COFLAGS Extra flags to the RCS co program. CPPFLAGS Extra flags to the C and Fortran preprocessor FFLAGS Extra flags to the Fortran compiler. GFLAGS Extra flags to the SCCS get program. LDFLAGS Extra flags to compilers when they invoke the linker, 'ld'. LFLAGS Extra flags to Lex. YFLAGS Extra flags to Yacc. PFLAGS Extra flags to the Pascal compiler. RFLAGS Extra flags to the Fortran compiler for Ratfor programs. LINTFLAGS Extra flags to lint.
(Z) Place Holder Template
Bah!
Testing:
Some ''whacky'' code. Next line.
Testing more:
Some whacky code. LINK last line
ok
