Hardware!
Goals from This Past Week
Team goals: Begin working on the physical design of the end product.
Personal goals: Find out how to run the H7 without being connected to a computer. This means finding a battery and making our program run by default (without requiring OpenMV).
Accomplishments
Power
Through looking at the documentation of the H7 board (from its product listing on OpenMV's website), I found that the regulator accepts an input between 2.5 to 5.5V, and that its configuration on the final board sets it up to output about 3.2V. Therefore, it should be sufficient to power it with a single 3.7V LiPo battery. It already has a JST set up to connect to a LiPo, so all I had to do was plug it in!
The image below shows all the hardware we need.
Code
Next, I had to get our SET solving script to run by default when the board starts up. I did some research, and the file system for the H7 is a bit weird. The quick summary is that I have to edit the "main.py" file that is on the board's base filesystem. Once I edit that file, the bootloader behind the scenes executes that file by default. That file is saved in the H7's non-volatile memory, so it runs every time the board starts up.
Online resources said that all external files need to be saved in the H7's base filesystem (not the SD card). However, I found that the H7 was able to utilize files both on the SD card and its base file system. So, all I have to do is plug the H7 into my computer and copy the required files into the device (either the H7 or SD card) that mounts. This allows the code to be broken up between multiple files so it all remains readable and organized.
Now for the behind-the-scenes filesystem information (if you are curious). The filesystem for the OpenMV board operates a bit differently from other systems. The H7 board supports an on-board SD card, but an external computer must interact with this SD card separately from the H7's memory. This confuses most modern computers, since they need to access two file systems (the H7 and the SD card) through one connection (the micro USB cable to the board). To solve this, when the H7 mounts to a computer's filesystem, it only mounts as either the H7 or SD card. If an SD card is present on the OpenMV board, it mounts as the SD card. If not, it mounts as the H7. To switch the device you interact with, you need to unplug the H7, insert or remove the SD card, and plug it back in.
So, to edit files on the base H7 flash, I have to plug the H7 into my computer without the SD card inserted. Then, I am free to edit the program files however I want so it runs the main program file whenever it starts up.
Code Refactoring
Since my main tasks went smoother than I anticipated, I spent some time refactoring Tyler's code to separate out the different phases of classification. At a high level, here is what I did:
Created functions to classify each attribute of a card
Moved classification code into a separate Python file
Documented all functions with function descriptions and annotations of argument/return types
Organized each file so that similar functions are located near each other
I organized the files we had into the following:
main.py --> main code
classifier.py --> classify the attributes of a card
image_processor.py --> process an image and return the 12 cards within it
set_detector.py --> detect sets given a list of 12 cards
matrix.py --> control the matrix
model.tflite --> compiled CNN for shape classification
CardTypes.py --> definition of types of cards and their attributes
Now, the main classification function's code became very simple:
thresh_img = threshold(card_img, verbose)
num_shapes = count_shapes(thresh_img, verbose)
shape_type = classify_shape(card_img, num_shapes)
color = classify_color(card_img, thresh_img, num_shapes, verbose)
fill = classify_fill(card_img, thresh_img, num_shapes, verbose)
return CardType(num_shapes, color, fill, shape_type)Similarly, the main code was also quite simple:
# Capture image from sensor
img = sensor.snapshot().save('frame.bmp') # takes image from sensor
# Convert image to grayscale
img_gray = img.copy(x_scale=1.0/scale_factor, y_scale=1.0/scale_factor)
# Get the 12 cards in the image
cards = process_img(img_gray, img, scale_factor, False)
# Identify Sets
sets = find_sets(cards)
# Light up matrix by iterating through each setHowever, when testing this code, the H7 consistently crashed. It seemed to be due to a memory issue — likely a hard fault — but OpenMV did not provide enough debug information to get into this error in much depth. Since the IDE uses Python to abstract away the lower level hardware details, I can’t dig too deeply into the debug information.
Since Tyler’s code works without refactoring it into functions, I decided that it is best to leave his code as is. Although that code is not as pretty as a what it could be (as ideally it is broken up between individual functions that each serve one function), it works. I think we can still leave the matrix and set solving code in separate files, but I will hold off on refactoring Tyler’s image processing code.
My theory is that passing image objects between functions is causing the crashes. Since these images can be large objects, passing them between the scopes of different functions may be causing some out of bounds memory access. This could either be because one function doesn’t have permission to access the image object or that images get garbage collected (deleted from memory) after a function returns because the H7 things it is no longer needed. Then, when a different function tries to access that image, it become an illegal access to memory.
Goals for Next Week
Team goals: Assemble a physical product and improve performance through testing.
Personal goals: Begin drafting final report
Comments