/contrib/famzah

Enthusiasm never stops


Leave a comment

Arduino ESP32 development under Linux

I am using a Windows desktop and I run Linux as a Virtualbox guest. ESP32 development under Windows is super easy to set up – you only need to install the Arduino IDE. Unfortunately, it really bugged me that compilation time is so slow. I’m not enough experienced and test a lot on the real hardware, and slow compilation really slows down the entire development process.

The Arduino IDE v2 has improved and supports parallel compilation within a submodule, but still it works slower than expected on my computer and is not ideally parallelized. Additionally, I noticed that some libraries are recompiled every time which is a huge waste of time (and resources) because libraries don’t change. Only my sketch source code changes.

I decided to see how compilation performs on Linux. The Arduino project has a command-line tool to manage, compile and upload code. The tool is still in active development but works flawlessly for me.

Here is how I installed it on my Linux desktop:

wget https://downloads.arduino.cc/arduino-cli/arduino-cli_latest_Linux_64bit.tar.gz
tar -xf arduino-cli_latest_Linux_64bit.tar.gz
mv arduino-cli ~/bin/
rm arduino-cli_latest_Linux_64bit.tar.gz

arduino-cli config init
vi /home/famzah/.arduino15/arduino-cli.yaml
  # board_manager -> additional_urls -> add "https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json"
  # library -> enable_unsafe_install -> set to "true" # so that we can install .zip libraries
arduino-cli core update-index
arduino-cli core install esp32:esp32
arduino-cli board listall # you must see lots of "esp32" boards now

Here is how to compile a sketch and upload it to an ESP32 board:

cd %the_folder_where_you_saved_your_Arduino_sketch%

arduino-cli compile --warnings all -b esp32:esp32:nodemcu-32s WifiRelay.ino --output-dir /tmp/this-sketch-tmp

arduino-cli upload -v --input-dir /tmp/this-sketch-tmp -p /dev/ttyUSB0 -b esp32:esp32:nodemcu-32s

arduino-cli monitor -p /dev/ttyUSB0 -c baudrate=115200

I have created a small script to automate this task which also uses a dynamically created temporary folder, in order to avoid race conditions and artefacts. The sketch folder on my Windows host is shared (read-only) with my Linux guest. I still write the sketch source code on my Windows machine using the Arduino IDE.

The ESP32 board is connected to my Windows host but I need to communicate with it from my Linux guest. This is done easily in Virtualbox using USB filters. A USB filter allows you to do a direct pass-through of the ESP32 serial port. It also works if you plug in the ESP32 board dynamically while the Linux machine is already running.


Leave a comment

ESP32 custom NodeMCU firmware build and flash

Here is my success story on how to compile and flash your own firmware for NodeMCU running on the ESP32 microcontroller. Although there is official documentation on how to achieve this, there were some minor issues that I stumbled across.

Using the Docker image is the recommended and most easy way to proceed. I didn’t have a Docker server so I started a cheap VPS from Digital Ocean. They have a one-click installer for Docker. Once the VPS machine is running, execute the following which covers the step “Clone the NodeMCU firmware repository” from the docs:

### log in via SSH to the Docker host machine

cd /opt

git clone --recurse-submodules https://github.com/nodemcu/nodemcu-firmware.git

cd nodemcu-firmware

git checkout dev-esp32
git submodule update --recursive
# https://github.com/marcelstoer/docker-nodemcu-build/issues/58
git submodule update --init

Then follow the instructions for the step “Build for ESP32” from the docs:

docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build configure-esp32
docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build build

Note: If you use the Heltec ESP32 WiFi Kit 32 with OLED, when configuring the firmware options you have to change the Main XTAL frequency from the default 40 Mhz to either “Autodetect” or 26 MHz, as explained here. I selected “Autodetect” (26 MHz) under “Component config” -> “ESP32-Specific” -> “Main XTAL frequency”.

For the impatient — most probably the only configuration you need to do is in the section “Component config” -> “NodeMCU modules”.

I encountered a compilation error which was caused by a defined-but-not-used function:

make[2]: Entering directory '/opt/nodemcu-firmware/build/modules'
CC build/modules/mqtt.o
CC build/modules/ucg.o
/opt/nodemcu-firmware/components/modules/ucg.c:598:12: error: 'ldisplay_hw_spi' defined but not used [-Werror=unused-function]
 static int ldisplay_hw_spi( lua_State *L, ucg_dev_fnptr device, ucg_dev_fnptr extension )
            ^
cc1: some warnings being treated as errors
/opt/nodemcu-firmware/sdk/esp32-esp-idf/make/component_wrapper.mk:285: recipe for target 'ucg.o' failed
make[2]: *** [ucg.o] Error 1
make[2]: Leaving directory '/opt/nodemcu-firmware/build/modules'
/opt/nodemcu-firmware/sdk/esp32-esp-idf/make/project.mk:505: recipe for target 'component-modules-build' failed
make[1]: *** [component-modules-build] Error 2

You can simply comment out the function in the firmware sources or configure to ignore the warning. Start a shell into the Docker image and edit the sources:

docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build bash
# you are in the Docker image now; fix the source code

vi /opt/nodemcu-firmware/components/modules/ucg.c # add the following (including the starting shebang sign):
#pragma GCC diagnostic ignored "-Wunused-function"

exit

# you are in the Docker host; redo the build as usual
docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build build

Because it took me a few attempts to configure everything properly and to finally build the image, I used the following alternative commands to make the process more easy:

docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build bash
# you are in the Docker image now
cd /opt/nodemcu-firmware
make menuconfig
make

Once the build completes it produces three files on the Docker host:

/opt/nodemcu-firmware/build/bootloader/bootloader.bin
/opt/nodemcu-firmware/build/NodeMCU.bin
/opt/nodemcu-firmware/build/partitions_singleapp.bin

Copy those three files on your local development machine where you are about to flash the NodeMCU controller. My first attempt was rather naive and the chip wouldn’t boot. It issued the following error on the USB console:

Early flash read error: "flash read err, 1000"

It turned out that I accidentally overwrote the bootloader and the partitions. Fortunately, the ESP32 official docs explain this in details here and here.

Here is what worked for me to flash my custom NodeMCU firmware into ESP32 from my Windows workstation:

esptool.py.exe --chip esp32 -p COM5 erase_flash

esptool.py.exe --chip esp32 -p COM5 write_flash 0x1000 .\bootloader.bin 0x8000 .\partitions_singleapp.bin 0x10000 .\NodeMCU.bin