Original Link : https://www.hackster.io/mhanuel/a-better-web-camera-powered-by-sony-d8d1d2
Story
About
This project has been inspired by all the Sony Spresense capabilities to use a new and innovative MCU that brings a powerful camera module to the developer community, the idea is to prove that by aim of the Spresense technology a new product can be developed in shorter time, using less hardware resources and with smarter and differentiating features compared to existing tools. The project was also of my interest in camera MCU designs where public previous experience work is available such as this (NXP) and this (WZ1320).
The Setup
The Sony Spresense board don’t have an Ethernet peripheral which is something we need for the web camera project. There might be many options out there to provide Ethernet connectivity but not all might serve for our purpose, been the speed and large amount of data transfer our makers that need to be maximized.
Based on previous experience projects, I have found Wiznet modules great for the Job, actually the experience with other technologies shows that the camera acquisition is more of the bottleneck to have a video stream than is the Ethernet connectivity when using Wiznet modules. Another advantage of using Wiznet modules was the easy implementation either from Hardware but more important from Firmware point of view. The last is specially true when using the Arduino IDE.
For the video stream we need the Spresense module with Main board and camera module, for the Wiznet module we have choose Wiz550io.
Hardware Hookup
The hardware is not complicated after sorting out a few issues explained below.
The Spresense SPI port that uses Arduino IDE is accessible using the Main board and has some limitation in speed compared to Wiz550io module which can be clock up to 80MHz. After starting testing the connectivity I noticed that it works intermittent, sometimes the library was not working at all, I play with speed and noticed that at low speed such 2 MHz it was more the times it works than at higher speed. This make me thing it was a hardware problem.
The Spresense documentation (section 4.7) mention that the I/O lines of SPI uses a level converter device, which I have check to be LSF0204x from TI. The device per se don’t seems to be the cause of problem, the problem seems to be an impedance matching between the W5500 device and the pull ups used, perhaps the current driving needs of Input lines of W5500 are not fulfilled. After using a couple of 74AHC1G125 single gate buffer device from TI for the Clock line (SCK) and Data line (MOSI), the issue seems to be solved at least to achieve a reliable operation at 14MHz.
The clock signal with and without the buffer driver can be seen below in a scope trigger capture. There is noticeable difference which might explain the issue. Unfortunately I haven’t a Quad gate buffer such SN74HC125 to test with all high speed lines to check how high the clock can be driven, but also it might not be good to test using a breadboard.
SPI4 is the port used by Arduino IDE and is located in the normal position you would find SPI for Arduino Uno R3. Signal line connections are similar to Arduino. One important difference is that Arduino wiznet library uses an I/O to drive the Chip Select pin. This is the suggested way to drive SPI for W5500 IC since CS signal should be active low during a transfer that might consist of several byte cycles. Because of this reason, we cannot connect the CS pin 10 of Spresense, normally labeled SS for arduino pinouts, as this pin is automatically handle by SPI4 peripheral of CXD5602 MCU, this is explained in section 3.8 of Spresense arduino developer guide.
Here is the wiz550io pinout for reference, but more info here.
The connections end up been:
Spresense MOSI (D11) -> Wiz550io J1 – pin 3
Spresense MISO (D12) -> Wiz550io J1 – pin 4
Spresense SCK (D13) -> Wiz550io J1 – pin 5
Spresense MOSI (D8) -> Wiz550io J1 – pin 6
Notice that we leave D10 open and use D8 for CS. This will require some change in arduino firmware library or sketch.
The firmware application needs and SD card as well as camera module connected. In order to have it close to Wiz550io the Main board is placed on top of breadboard as shown below.
Firmware
The final firmware has join several parts from different projects ported to Spresense Arduino core. The Spresense core is fully functional with Arduino IDE but that doesn’t mean all the existing libraries will compile or work out of the Box. That was the case with Arduino Ethernet library.
Ethernet Library Firmware
Original library can be found here. There wasn’t many changes to make Spresense work with this library, but enough to delay your anxious willing to do some tests after the really simple hookup done before. A list of changes is explained below
- CS need to change – This can be done using library call in sketch but for safety I also change the default in case User miss it.
- Enable the use of large buffers by un-commenting ETHERNET_LARGE_BUFFERS definition.
- The Spresense SPI transfer method has somehow different prototype, so I adjust it accordingly. Look for #ifdef SPI_HAS_TRANSFER_BUF in sources.
- Some cast issue with IPAddress. Look for IPAddress((uint32_t)0ul) in source.
All the above can be seen on this commit under github. Just clone my fork of Ethernet library found here and replace your local copy that comes with Arduino IDE.
The First Image
As always I try to go from basic to more complex project by introducing small changes to a working baseline. In this case with Ethernet library working, I look for a way to transmit an image and shows in a HTTP browser.
First thing that become clear when using Spresense was that jpeg option was not appropriate, in other projects that I have the opportunity to work with jpeg compression, different reasons lead me to take that solution, for example the small size of image after jpeg compression and the slow image capture from camera are among others the more remarkable ones.
In the case of Spresense, the jpeg is used only for still image acquisition, which is really slow. Also the jpeg image is high resolution image which means we are not getting the jpeg image of an original image of QVGA resolution for example.
So the only way here to go was using directly other format such YUV or BMP. The latter will allow us to do it simple, at keast to embed it in HTTP web page. But also the YUV available is YUV422, which is same size of RGB565.
Because I always feel like dummy if I realize what I am doing is like reinventing the wheel, I look for similar work that I can use. In this case I found this excellent work for esp32, you can take a look at the sources here.
I decide to port the code for spresense + wiznet. So basically only the data transfer will be of importance. But actually I go ahead one step with above project, If you look at it, the project uses websockets, which we haven’t yet. The starting point would be a project from another author here with sources here. The last uses ESP32 but no stream if I remember correctly, the image is just loaded on HTML when ESP32 serve the page. For our starting purposes is just great.
The section of code that serves the web pages would require no change. I then ported the OV7670 and RGB565 image section of code to what our spresense platform provides. A working code can be found on this gist. You will also need the BMP.h header which has no change from original project and found here. If you load this project onto Spresense with camera and Wiznet module connected you should see a web page with only an image capture of camera module.
How It Works
Let’s discuss some details about how it works since this will be the basic of our more advance camera streaming firmware.
First we configure the camera module for 30 FPS and YUV422. The QVGA seems to be the only option available under streaming mode, I have found other resolution to give some issues. Then we call startStreaming function but in disable mode, that is
theCamera.begin(1,
CAM_VIDEO_FPS_30,
CAM_IMGSIZE_QVGA_H,
CAM_IMGSIZE_QVGA_V,
CAM_IMAGE_PIX_FMT_YUV422);
theCamera.startStreaming(false, CamCB);
The reason for this is because we will trigger the stream when a new query came from HTML server.
A variable called camStreamRdy will be set to true once the capture image is ready, this will happen inside the callback function void CamCB(CamImage img).
We then check for this variable flag and when done just disable the streaming.
if(currentLine.endsWith("GET /camera"))
{
imageBuf = NULL;
theCamera.startStreaming(true, CamCB);
for(int j=0; j < 1000; j++){
delay(1);
if(camStreamRdy == true){
theCamera.startStreaming(false, CamCB);
break;
}
}
if (camStreamRdy == true)
{
camStreamRdy = false;
if(imageBuf != NULL){
client.println("HTTP/1.1 200 OK");
client.println("Content-Type: image/bmp");
client.println();
BMP::construct16BitHeader(bmpHeader, imageWidth, imageHeight);
for(int i = 0; i < BMP::headerSize; i++){
client.write(bmpHeader[i]);
}
size_t len = imageSize;
int i = 0;
while(len > 0){
if(len > WIZSSIZE){
client.write(&imageBuf[i], WIZSSIZE);
i += WIZSSIZE;
len -= WIZSSIZE;
}else{
client.write(&imageBuf[i], len);
break;
}
}// while
}//if image
}//if stream ready
}//if HTML GET camera
When the HTML server receives a GET query with the URL of /camera, the stream is enable and we keep checking the flag for culmination. If there is a valid image (by checking the image buffer pointer), then The HTML server replay with 200 OK and send the BMP header along the RGB565 image. The image conversion is done under the image callback function, but also the image buffer pointer and size details, that is
img.convertPixFormat(CAM_IMAGE_PIX_FMT_RGB565);
imageBuf = img.getImgBuff();
imageSize = img.getImgSize();
imageHeight = img.getHeight();
imageWidth = img.getWidth();
The importance of this has to do with how we send the BMP image response. The BMP header need the Height and Width dimensions. But also the image data is send in chunks of bytes of size WIZSSIZE. The WIZSSIZE is a fix define in our code, basically I just found the Wiznet library write function would not work as expected when it’s called with data sizes bigger than the buffer sizes of socket. That’s the reason we send chunks using a while loop that exits when total length has been sent, that is
while(len > 0){
if(len > WIZSSIZE){
client.write(&imageBuf[i], WIZSSIZE);
i += WIZSSIZE;
len -= WIZSSIZE;
}else{
client.write(&imageBuf[i], len);
break;
}
}
One more thing to pay attention here is that the library should use the same socket buffer as we define in our sketch, i.e. if our sketch defines WIZSSIZE as
#define WIZSSIZE 8192
We have to initialize our socket to have 8K in size. For this you have to remember that W5500 can have up to 32K of total memory that might divide equally among up to 8 sockets. In our example, to have 8K memory in a socket we have to limit the total number of available sockets by 2. This is done in Ethernet.h file, look for a line like the following and change it accordingly.
#define MAX_SOCK_NUM 4
Streaming Camera
The previous section described how to gather an image using HTML and BMP as type of file. This was a step I did to verify the feasibility of handling a streaming later on. But streaming doing plain HTML and using javascript reload or similar is not good idea. The other project that uses WebSockets comes very handy for this purpose.
The main problem is to have a WebSocket Arduino library working for Spresense. I pick up the same library used by the aforementioned project that can be found here. The first issue you will find to make this library work for Spresense core is that similar to the AVR/ATmega core, std namespace of C++ seems as not supported, so you will get a bunch of compiler errors.
To overcome this, just use the Atmega branch. Thou compilation pass, the library has some issue when establishing a connection, more than one problem seems to be the culprit. The way the library handles new clients is somehow broken for Spresense, not totally sure the exact cause but the function clientIsConnected returns true the first time after server is listening, but in subsequent calls it return false even when the client is still connected.
I decide to rewrite this section somehow by adding a new function with a name that should be familiar under Arduino libraries, that is process. This way the sketch just have to call process instead of loop function. The process function will only handle new clients when the server available method returns true, this happen after a connect, because server is listening. Later on after the connection is established, a client instance is assigned to the new connection and keep in a list which handleClientData function checks in a loop for available data.
void WebSocketsServer::process(void) {
EthernetClient tmp;
tmp = _server->available();
if(tmp){
// we have a new client
// DEBUG_WEBSOCKETS("[WS-Client] process a new client!n");
newClient(&tmp);
}
else{
handleClientData();
}
}
I added a function called newClient which does a similar task as handleNewClients. The difference is the way it save the client instance from the server, with this modification, the function handleClientData do not require any change. You can clone the Atmega branch of my Fork here.
One important modification is the function sendFrame in WebSockets.cpp file. As happen with the Ethernet library, here we have to send the data in chunks of maximum size given by W5500 number of sockets in use. A similar while loop is implemented.
if(payload && length > 0) {
size_t len = length;
int i = 0;
while(len > 0){
if(len > WIZSSIZE){
client->tcp->write(&payload[i], WIZSSIZE);
i += WIZSSIZE;
len -= WIZSSIZE;
}else{
client->tcp->write(&payload[i], len);
break;
}
}
}
You can find all the changes by looking at the commit details here.
If you check the stream project based on esp32, I modify it to suit our Spresense platform. Both the sketch and the html (in the form of a header file) has been modified, just check them out. I will continue and leave the demo time for the advance web server, which uses almost same html and websocket logic for video stream. Before jumping into a more advance web camera server, let’s discuss how the camera streaming works using websockets.
All the websocket logic is contained in a function called webSocketEvent, this function is called whenever a new websocket event is present such connection or incoming message.
void webSocketEvent(uint8_t num, WStype_t typein, uint8_t * payload, size_t payloadlength) { // When a WebSocket message is received
int blk_count = 0;
char canvas_Q_VGA[] = "canvas-Q-VGA";
char ipaddr[26];
switch (typein) {
case WStype_DISCONNECTED: // if the websocket is disconnected
Serial.printf("[%d]", num);
Serial.print(" Disconnected!n");
break;
case WStype_CONNECTED: { // a new websocket connection is established
webSocket.sendBIN(0, &ip_flag, 1);
// parse local IP address
sprintf(ipaddr, "%d.%d.%d.%d", localip[0], localip[1], localip[2], localip[3]);
webSocket.sendTXT(0, (const char *)ipaddr);
}
break;
case WStype_TEXT: // if new text data is received
if (payloadlength == sizeof(canvas_Q_VGA)-1) {
if (memcmp(canvas_Q_VGA, payload, payloadlength) == 0) {
webSocket.sendBIN(0, &end_flag, 1);
}
}
// capture image from camera stream
theCamera.startStreaming(true, CamCB);
for(int j=0; j < 1000; j++){
delay(1);
if(camStreamRdy == true){
break;
}
}
if (camStreamRdy == true)
{
camStreamRdy = false;
theCamera.startStreaming(false, CamCB);
if(imageBuf != NULL){
size_t len = imageSize/2;
blk_count = 2;
int j = 0;
for (int i=0; i<blk_count; i++) {
if (i == 0) {
webSocket.sendBIN(0, &start_flag, 1);
}
if (i == blk_count-1) {
webSocket.sendBIN(0, &end_flag, 1);
}
webSocket.sendBIN(0, &imageBuf[j], len);
j += len;
}
} //imageBuf != NULL
}
break;
case WStype_ERROR: // if new text data is received
Serial.println("Error");
default:
Serial.print("WStype ");
Serial.print(typein);
Serial.println(" not handled n");
}
}
The original code can use different image sizes. To my best understanding the camera stream (a Preview image) for spresense currently only support QVGA, for this reason I have remove any code that implements different image size. The first time a new websocket connection is established, the server sends it’s IP address and the client then should send a string with image size name, such “canvas-Q-VGA” which will setup the start of new image acquisition. This is done very similar to the BMP example. After image is acquired, the stream is stopped meantime the image is sent to client. Original code requires the image to be sent in two parts or chunks, but with a “flag” that identify the first or last part of image data. For the QVGA image size it means 76800 bytes are sent along the starting flag 0xAA and the remaining 76800 bytes along end of frame flag 0xFF.
The HTML javascript code will handle the data and convert to RGB565 for displaying. After websocket is connected (see the onopen code), a capture is initiated but just after the canvas to hold the image is created. This is done in a function called capture.
function capture(canvasid)
{
if (ws.readyState != 1) {
// alert("ws.readyState " + ws.readyState);
return;
}
reset();
gcanvasid = canvasid;
canvas = document.getElementById(canvasid);
ctx = canvas.getContext('2d');
if (canvasid.indexOf("canvas-Q-VGA", 0) != -1) {
xres = 320;
yres = 120;
} else {
return;
}
imgData = ctx.createImageData(canvas.width, canvas.height);
for (var i=0;i<imgData.data.length;i+=4)
{
imgData.data[i+0]=0xCC;
imgData.data[i+1]=0xCC;
imgData.data[i+2]=0xCC;
imgData.data[i+3]=255;
}
ctx.putImageData(imgData, canvas.width, canvas.height);
ws.send(canvasid);
}
the canvasid variable is actually the string “canvas-Q-VGA”.
The onmessage callback basically will keep track of which segment has been received and upon completion will call display function.
ws.onmessage = function (evt) {
var arraybuffer = evt.data;
// console.log('onmessage receive len: ' + arraybuffer.byteLength);
if (arraybuffer.byteLength == 1) {
flag = new Uint8Array(evt.data); // Start Flag
if (flag == 0xAA) {
// console.log("Start Block");
ln = 0;
}
if (flag == 0xFF) {
// console.log("Last Block");
}
if (flag == 0x11) {
// console.log("Camera IP");
}
} else {
if (flag == 0x11) {
camera_ip = evt.data;
document.getElementById("eth-ip").innerText = camera_ip;
flag = 0;
} else {
var bytearray = new Uint8Array(evt.data);
display(bytearray, arraybuffer.byteLength, flag);
}
}
}; //ws.onmessage
The importance of the start and end flag can be seen on the display function.
function display(pixels, pixelcount, flag) {
var i = 0;
for(y=0; y < yres; y++) {
for(x=0; x < xres; x++)
{
i = (y * xres + x) << 1;
pixel16 = (0xffff & pixels[i]) | ((0xffff & pixels[i+1]) << 8);
imgData.data[ln+0] = ((((pixel16 >> 11) & 0x1F) * 527) + 23) >> 6;
imgData.data[ln+1] = ((((pixel16 >> 5) & 0x3F) * 259) + 33) >> 6;
imgData.data[ln+2] = (((pixel16 & 0x1F) * 527) + 23) >> 6;
imgData.data[ln+3] = (0xFFFFFFFF) & 255;
ln += 4;
}
}
if (flag == 0xFF) { // last block
ln = 0;
ctx.putImageData(imgData,0,0);
ws.send("next-frame");
}
}
This function will convert the RGB565 image to RGB888. The ln variable keeps track of RGB888 pixel to write, when end of flag is received the image is displayed by doing
ctx.putImageData(imgData,0,0);
A new image is immediately requested by sending “next-frame” string to server.
Advance Web Camera
The Server code works as expected, but a couple of things can be enhanced. The html server file is stored in a C header file, which is really annoying to modify when needed. Also, anything like adding a button on web page becomes difficult because there HTML server is really basic.
The Advance Web camera Server will overcome this limitations by serving files from SD card and implementing an HTML server very handy.
The spresense make svery easy to add SD card to our project. Any modification to a file in SD card is also very easy since the spresense can be mounted as a USB Mass Storage Controller device, and the best is that we can control how and when to mount the MSC device.
The first thing to do when running code will be to handle MSC device, the following snippet is executed under setup sketch function.
Serial.begin(115200);
Serial.println("Setting up SD card...");
if (!SD.begin()) {
has_filesystem = false;
Serial.println("SD card is not present");
}
Serial.println("Enter any character to Start USB MSC");
bool startMSC = false;
int starttime = millis();
while (!Serial.available()){
if((millis() - starttime) > 3000){
break;
}
}
if(Serial.available()){
while (Serial.available()) {
Serial.read(); // dummy read to empty input buffer
}
startMSC = true;
}
if(startMSC){
Serial.println("Starting USB MSC");
// Start USB Mass Storage
if (SD.beginUsbMsc()) {
Serial.println("UsbMsc connect error");
}
Serial.println("Finish USB MSC? (y/n)");
while (true) {
while (!Serial.available());
if(Serial.read() == 'y'){
while (Serial.available()) {
Serial.read(); // dummy read to empty input buffer
}
break;
}
}
// Finish USB Mass Storage
if (SD.endUsbMsc()) {
Serial.println("UsbMsc disconnect error");
}
Serial.println("<<< Finish USB Mass Storage Operation");
delay(100);
}
Basically, we wait a few seconds before jumping into normal run mode only if user don’t enter any character on the serial console, otherwise we mount the MSC device and wait for the user to indicate when ready to resume operation.
In order to serve HTML files from SD card and have other nifty aids in our server code, I have ported the TinyWebServer code to Spresense. You can find my Forked project here ready to be use for Spresense.
Two main modifications are good to mention. First the SD card is adapted to use the spresense core functions to access Sd card file system.
Second, a new namespace is provided, that is TinyWebFormHandler, this allow us to have a Form that can be submitted. If you have experience with TinyWebServer code, you can check easily the differences by looking at this commit.
Now you are ready to load the advance web camera sketch onto spresense and load the webcam.htm file on a SD card to run the demo code.
Make sure you open the serial console and see the following
Enter any character to Start USB MSC
Size Filename
---- --------
12 /System Volume Information/WPSettings.dat
76 /System Volume Information/IndexerVolumeGuid
9285 /webcam.htm
153738 /Media/minion.bmp
WebSockets Camera Demo
Prepare camera
Streaming starts with New Websocket
Set Auto white balance Default parameter
Setting up the Wiz550io Ethernet Module
Web server starting...
Ready to accept HTTP requests at 192.168.1.109
WebSocket server started at port 9001
Now you can point your browser to 192.168.1.109/webcam.htm, just pay attention to the IP address assigned by your DHCP server. You can also use a static IP, just change Ethernet begin call in that case.
A basic HTML form has been implemented to change the camera filter.
Here is a simple demo for filter change.
I haven’t measure the FPS of streaming but it’s not that bad, at least comparable with my baby web cam is really good.
Final Thoughts
This project put into test the extensive features of Spresense for image acquisition, demonstrate how powerful and versatile it is. A reliable and working demo has been shown possible, the project creativity comes from the envisioned idea to make a streaming camera possible using all it’s features and other existing projects as well, solving bugs and porting code proves to be a smarter approach to a faster product as well.
Because of time constraint the camera module has some missing features such two way audio and PTZ. Nonetheless, the foundations are set to allow further development. Specially the connectivity and advance server features make easier the improvements. For example, a PTZ camera can be easily implement (speaking about the electronics) using web forms to receive the movement commands.
Happy Spresense web streaming!
COMMENTS