Testing C / C ++ Projects Using Python

Introduction


The ability to integrate Python and C / C ++ is well known. Typically, this technique is used to speed up Python programs or to fine tune C / C ++ programs. I would like to highlight the possibility of using python to test C / C ++ code in the IDE without supporting the test organization system in the IDE. From my point of view, it is advisable to apply it in the field of software development for microcontrollers.

You can talk a lot about the need for tests in projects, I assume that tests help me develop the functionality of the program. And after the completion of the project, after some time, they help to understand it and protect from errors.

When developing programs for microcontrollers, I was faced with the lack of standard input / output (of course, you can redefine the input and output functions in the simulator, output data via UART - but often UART is already involved, and the simulator does not always work correctly) and there are big risks to disable hardware erroneous business logic. At the development stage, I implemented individual projects that tested parts of the program and then I was responsible for launching all test applications after making changes. Of course, all this can be automated. This can work, but I found a better way.

Description of the technique


It is possible to use python (namely ctypes) to cover tests of individual modules of a C / C ++ project. The essence of the technique is to create isolated parts that implement part of the functionality in the form of dynamically linked libraries (dlls), feed data and control the result. Python is used as a "binding". This technique does not imply changes to the code of the tested application.

To test individual pieces of code, you may need to create an additional file with / c ++ - the “adapter”, to combat the naming of overloaded functions (questions on the naming of exported functions are discussed in detail in habrahabr.ru/post/150327 ) or with functionality having complex dependencies and hard to implement in the "ideology" dll.

Necessary software environment


This technique implies the ability to compile individual parts of the program from the command line. So we need a c / c ++ compiler, and a python interpreter. For example, I use GCC (for windows - MinGW (MinGw www.mingw.org ), python ( www.python.org ), but linux distributions usually have everything that is installed by default).

Usage example


To illustrate this technique, I will give the following example:
source project:
file structure:

  + --- Project |  Makefile  
    + --- src  
        + --- api | ApiClass.cpp | ApiClass.h | ApiFunction.cpp | ApiFunction.h |  
        \ --- user main.cpp 

Project Files:

File ApiFunction.cpp
#include "ApiFunction.h"

#include <cstring>

int apiFunction(int v1, int v2){
	return v1*v2;
}
void apiFunctionMutablePointer(double * value){


	* value = *value * 100;
}


Data apiFunctionGetData(){

	Data dt;

	dt.intValue = 1;
	dt.doubleValue = 3.1415;
	dt.ucharValue = 0xff;

	return dt;
}

Data GLOBAL_DATA;


Data * apiFunctionGetPointerData(){

	GLOBAL_DATA.intValue = 1*2;
	GLOBAL_DATA.doubleValue = 3.1415*2;
	GLOBAL_DATA.ucharValue = 0xAA;

	return &GLOBAL_DATA;
}

void apiFunctionMutablePointerData(Data * data){
	data->intValue = data->intValue * 3;
	data->doubleValue = data->doubleValue *3;
	data->ucharValue = data->ucharValue * 3;
}


BigData apiFunctionGetBigData(){
	BigData bd;

	bd.iv = 1;
	bd.v1 = 2;
	bd.v2 = 3;
	bd.v3 = 4;
	bd.v4 = 5;

	std::memset(bd.st,0,12);
	std::memmove(bd.st,"hello world",12);

	return bd;
}




File ApiFunction.h
#ifndef SRC_API_APIFUNCTION_H_
#define SRC_API_APIFUNCTION_H_


#ifdef __cplusplus
extern "C" {
#endif

int apiFunction(int v1, int v2);

void apiFunctionMutablePointer(double * value);

struct Data{
	int intValue;
	double doubleValue;
	unsigned char ucharValue;
};


struct BigData{
	int iv;
	int v1:4;
	int v2:4;
	int v3:8;
	int v4:16;

	char st[12];

};


Data apiFunctionGetData();

Data * apiFunctionGetPointerData();

void apiFunctionMutablePointerData(Data * data);

BigData apiFunctionGetBigData();


#ifdef __cplusplus
}
#endif

#endif




File ApiClass.cpp
#include "ApiClass.h"
#include <iostream>

ApiClass::ApiClass():value(0) {
	std::cout<<std::endl<<"create ApiClass value = "<<value<<std::endl;
}
ApiClass::ApiClass(int startValue):
		value(startValue){
	std::cout<<std::endl<<"create ApiClass value = "<<value<<std::endl;
}

ApiClass::~ApiClass() {
	std::cout<<std::endl<<"delete ApiClass"<<std::endl;
}

int ApiClass::method(int vl){
	value +=vl;
	return value;
}




File ApiClass.h
#ifndef SRC_API_APICLASS_H_
#define SRC_API_APICLASS_H_

class ApiClass {
public:
	ApiClass();
	ApiClass(int startValue);
	virtual ~ApiClass();

	int method(int vl);


private:
		int value;
};

#endif




File main.cpp
#include <iostream>

#include "ApiFunction.h"
#include "ApiClass.h"

int main(){
	std::cout<<"start work"<<std::endl;
	std::cout<<"=============================================="<<std::endl;
	std::cout<<"call apiFunction(10,20) = "<<apiFunction(10,20)<<std::endl;
	std::cout<<"call apiFunction(30,40) = "<<apiFunction(30,40)<<std::endl;

	std::cout<<"=============================================="<<std::endl;
	ApiClass ac01;
	std::cout<<"call ac01.method(30) = "<<ac01.method(30)<<std::endl;
	std::cout<<"call ac01.method(40) = "<<ac01.method(40)<<std::endl;

	std::cout<<"=============================================="<<std::endl;
	ApiClass ac02(10);
	std::cout<<"call ac02.method(30) = "<<ac02.method(30)<<std::endl;
	std::cout<<"call ac02.method(40) = "<<ac02.method(40)<<std::endl;
}




makefile
FOLDER_EXECUTABLE = bin/
EXECUTABLE_NAME = Project.exe

EXECUTABLE = $(FOLDER_EXECUTABLE)$(EXECUTABLE_NAME)
FOLDERS = bin bin/src bin/src/api bin/src/user
SOURSES = src/user/main.cpp src/api/ApiClass.cpp src/api/ApiFunction.cpp

CC = g++
CFLAGS = -c -Wall -Isrc/helper -Isrc/api 
LDFLAGS = 
OBJECTS = $(SOURSES:.cpp=.o)
OBJECTS_PATH = $(addprefix $(FOLDER_EXECUTABLE),$(OBJECTS))

all: $(SOURSES) $(EXECUTABLE)

$(EXECUTABLE): $(OBJECTS)	
	$(CC) $(LDLAGS) $(OBJECTS_PATH) -o $@
	
.cpp.o:
	mkdir -p $(FOLDERS)
	$(CC) $(CFLAGS) $< -o $(FOLDER_EXECUTABLE)$@

clean:
	rm -rf $(OBJECTS) $(EXECUTABLE) 



To cover the tests, add the test folder to the project folder. In this folder, we will have everything related to testing.

For convenience, we will create the helpers folder in the test folder (python package do not forget to create the __init__.py file inside) - it will have auxiliary functions common to all tests.
Helper functions from helpers package:

File callCommandHelper.py
import subprocess

class CallCommandHelperException(Exception):
    pass    

def CallCommandHelper(cmd):
    with subprocess.Popen(cmd, stdout=subprocess.PIPE,shell=True) as proc:
        if proc.wait() != 0:            
            raise CallCommandHelperException("error :" +cmd)




File creteDll.py
import os
from helpers import callCommandHelper

def CreateDll(folderTargetName, fileTargetName,fileSO):
    
    templateCompill = "g++  {flags}  {fileSourse} -o {fileTarget}"
    templateLinc    = "g++  -shared {objectfile} -o {fileTarget}"


    if os.path.exists(folderTargetName) == False:
        os.makedirs(folderTargetName)
    
#---------------delete old version-----------------------------------
    if os.path.exists(fileTargetName):
        os.remove(fileTargetName)        
    for fso in fileSO:
        if os.path.exists(fso["rezultName"]):
            os.remove(fso["rezultName"])            
#---------------compil -----------------------------------------------    
    for filePair in fileSO:
        fileSourseName  =  filePair["sourseName"]
        fileObjecteName = filePair["rezultName"]
        flagCompil = filePair["flagsCompil"]
        cmd = templateCompill.format(
            fileSourse = fileSourseName,
            flags      = flagCompil, 
            fileTarget = fileObjecteName)        
        
        callCommandHelper.CallCommandHelper(cmd)   
#---------------linck-----------------------------------------------
    fileObjectName = " "
    for filePair in fileSO:
        fileObjectName = fileObjectName + filePair["rezultName"]+" "
    
    
    cmd = templateLinc.format(
        objectfile = fileObjectName,
        fileTarget = fileTargetName)    
    
    callCommandHelper.CallCommandHelper(cmd)    
#====================================================== 


Note: If you use a compiler other than gcc, then you need to correct the name of the programs in the variables templateCompill and templateLinc.

In the creteDll.py file, all the magic of creating a test dll happens. I just create commands for compiling and linking (building) the dll for the operating system I use. As an option, it is possible to create a makefile template and substitute file names there, but it seemed easier to me. (in general, as I understand it, all the testing work can be submitted to the makefile, but it seems complicated to me, and projects created in keil or in other IDEs are not always built on makefile).

This completes all the preparation, now we can start testing.

Easy test creation


Consider the option of creating a test without using an adapter.

We test the functions from the files АpiFunction.h / АpiFunction.cpp.

Create a folder in the test folder for ApiFunctionTest for the created dll. Create a Python file for the test using the unittest module. In the setUpClass method, the dll is created, the functions are loaded and "tuned". And later we need to write standard methods for testing.

File apiFunctionTest.py
import os
import ctypes

from helpers import creteDll

import unittest

class Data(ctypes.Structure):
    _fields_ = [("intValue",ctypes.c_int),("doubleValue",ctypes.c_double),("ucharValue",ctypes.c_ubyte)]

class BigData(ctypes.Structure):
    _fields_ = [("iv",ctypes.c_int),
                ("v1",ctypes.c_int,4),
                ("v2",ctypes.c_int,4),
                ("v3",ctypes.c_int,8),
                ("v4",ctypes.c_int,16),
                ("st",ctypes.c_char*12)]

class ApiFunctionTest(unittest.TestCase):
    @classmethod
    def setUpClass(self):
       

        folderTargetName = os.path.join(os.path.dirname(__file__),"ApiFunctionTest")
       
        fileSO =  [
                    {"sourseName":"../src/api/ApiFunction.cpp",
                    "flagsCompil":"-Wall -c -fPIC",
                    "rezultName" :os.path.join(folderTargetName,"ApiFunction.o")}
                  ]
                    
        fileTargetName = os.path.join(folderTargetName,"ApiFunction.dll")
                
        #=============================================================
        creteDll.CreateDll(folderTargetName, fileTargetName, fileSO)
        
        
        lib = ctypes.cdll.LoadLibrary(fileTargetName)
    
        self.apiFunction = lib.apiFunction
        self.apiFunction.restype = ctypes.c_int

        self.apiFunctionMutablePointer = lib.apiFunctionMutablePointer
        self.apiFunctionMutablePointer.argtype  = ctypes.POINTER(ctypes.c_double)
        
        
        self.apiFunctionGetData = lib.apiFunctionGetData
        self.apiFunctionGetData.restype = Data
        
        
        self.apiFunctionGetPointerData = lib.apiFunctionGetPointerData
        self.apiFunctionGetPointerData.restype = ctypes.POINTER(Data)
         
        self.apiFunctionMutablePointerData = lib.apiFunctionMutablePointerData
        self.apiFunctionMutablePointerData.argtype  = ctypes.POINTER(Data)
       
        
        self.apiFunctionGetBigData = lib.apiFunctionGetBigData
        self.apiFunctionGetBigData.restype = BigData
        
       
        
    def test_var1(self):
        self.assertEqual(self.apiFunction(10,20), 200,'10*20 = 200')


    def test_var2(self):
        self.assertEqual(self.apiFunction(30,40), 1200,'30*40 = 1200')


    def test_var3(self):
        vl = ctypes.c_double(1.1)        
        self.apiFunctionMutablePointer(ctypes.pointer(vl) )
        self.assertEqual(vl.value, 110.00000000000001,'vl != 110')
        
    def test_var4(self):
        data = self.apiFunctionGetData()
        self.assertEqual(data.intValue, 1,'data.intValue != 1')
        self.assertEqual(data.doubleValue, 3.1415,'data.doubleValue != 3.1415')
        self.assertEqual(data.ucharValue, 0xff,'data.ucharValue != 0xff')
               
    def test_var5(self):
        pointerData = self.apiFunctionGetPointerData()    
        
        self.assertEqual(pointerData.contents.intValue, 1*2,'data.intValue != 1*2')
        self.assertEqual(pointerData.contents.doubleValue, 3.1415*2,'data.doubleValue != 3.1415 * 2')
        self.assertEqual(pointerData.contents.ucharValue, 0xAA,'data.ucharValue != 0xAA')
       

        
    def test_var5(self):
        pointerData = ctypes.pointer(Data())
        pointerData.contents.intValue = ctypes.c_int(10)
        pointerData.contents.doubleValue = ctypes.c_double(20)
        pointerData.contents.ucharValue = ctypes.c_ubyte(85)
       
        self.apiFunctionMutablePointerData(pointerData)
                 
        self.assertEqual(pointerData.contents.intValue, 30,'data.intValue != 30')
        self.assertEqual(pointerData.contents.doubleValue, 60,'data.doubleValue != 60')
        self.assertEqual(pointerData.contents.ucharValue, 0xff,'data.ucharValue != 0xff')
             
    def test_var6(self):
        
        bigData = self.apiFunctionGetBigData()
        st = ctypes.c_char_p(bigData.st).value
                
        self.assertEqual(bigData.iv, 1,'1')
        self.assertEqual(bigData.v1, 2,'2')
        self.assertEqual(bigData.v2, 3,'3')
        self.assertEqual(bigData.v3, 4,'4')
        self.assertEqual(bigData.v4, 5,'5')
        
        
        self.assertEqual(st in b"hello world",True,'getting string')


Note: If you use a compiler other than gcc, you need to fix the line with the flagsCompil key.

As you can see, there is no need for any additional actions for testing. We are limited only by the imagination of creating test scripts. In this example, the possibilities of transferring to various functions and obtaining various data types from them are demonstrated (this is described in more detail in the ctypes documentation ).

Creating a test using the “adapter”


Consider the option of creating a test using an “adapter”.

Test the ApiClass class from the ApiClass.h / ApiClass.cpp files. As you can see, this class has several options for creating; it also saves state between calls. In the test folder, create a folder for ApiClassTest for the created dll, and the “adapter” - ApiClassAdapter.cpp.

File ApiClassAdapter.cpp
#include "ApiClass.h"

#ifdef __cplusplus
extern "C" {
#endif

ApiClass * pEmptyApiClass = 0;
ApiClass * pApiClass = 0;

void createEmptyApiClass(){
	if(pEmptyApiClass != 0){
		delete pEmptyApiClass;
	}
	pEmptyApiClass = new ApiClass;
}
void deleteEmptyApiClass(){
	if(pEmptyApiClass != 0){
		delete pEmptyApiClass;
		pEmptyApiClass=0;
	}
}

void createApiClass(int value){
	if(pApiClass != 0){
		delete pApiClass;
	}
	pApiClass = new ApiClass(value);
}
void deleteApiClass(){
	if(pApiClass != 0){
		delete pApiClass;
		pApiClass=0;
	}
}

int callEmptyApiClassMethod(int vl){
	return pEmptyApiClass->method(vl);
}

int callApiClassMethod(int vl){
	return pApiClass->method(vl);
}


#ifdef __cplusplus
}
#endif


As you can see, the “adapter” simply wraps the ApiClass class calls for the convenience of calls from python.

To test this class, create the apiClassTest.py file.

ApiClassTest.py file
import os
import ctypes

from helpers import creteDll

import unittest

class ApiClassTest(unittest.TestCase):
    @classmethod
    def setUpClass(self):

                
        folderTargetName = os.path.join(os.path.dirname(__file__),"ApiClassTest")
        
        
        fileSO =  [
                    {
                    "sourseName":os.path.abspath("../src/api/ApiClass.cpp"),
                    "flagsCompil":"-Wall -c -fPIC",
                    "rezultName" :os.path.join(folderTargetName,"ApiClass.o")
                    },
                    {
                    "sourseName":os.path.join(folderTargetName,"ApiClassAdapter.cpp"),
                    "flagsCompil":"-Wall -c -fPIC -I../src/api",
                    "rezultName" :os.path.join(folderTargetName,"ApiClassAdapter.o")
                    }
                   ]
                   
        fileTargetName = os.path.join(folderTargetName,"ApiClass.dll")
        #======================================================
        creteDll.CreateDll(folderTargetName, fileTargetName, fileSO)    
#======================================================
        lib = ctypes.cdll.LoadLibrary(fileTargetName)
    
        self.createEmptyApiClass = lib.createEmptyApiClass        
        self.deleteEmptyApiClass = lib.deleteEmptyApiClass        
    
        self.callEmptyApiClassMethod = lib.callEmptyApiClassMethod
        self.callEmptyApiClassMethod.restype = ctypes.c_int
        
        self.createApiClass = lib.createApiClass        
        self.deleteApiClass = lib.deleteApiClass        
    
        self.callApiClassMethod = lib.callApiClassMethod
        self.callApiClassMethod.restype = ctypes.c_int
        
    
    def tearDown(self):
        self.deleteEmptyApiClass()
        self.deleteApiClass()
    
        
    def test_var1(self):
        self.createEmptyApiClass()
        self.assertEqual(self.callEmptyApiClassMethod(10), 10,'10+0 = 10')
        self.assertEqual(self.callEmptyApiClassMethod(20), 30,'20+10 = 30')


    def test_var2(self):
        self.createApiClass(100)
        self.assertEqual(self.callApiClassMethod(10), 110,'10+100 = 110')
        self.assertEqual(self.callApiClassMethod(20), 130,'20+110 = 130')


Here you should pay attention to the tearDown method, in it, after each test method, the objects created in the dll are deleted to prevent memory leaks (in this context, it does not really matter).

Well, the union of all the tests in the file TestRun.py

file TestRun.py
import unittest
loader = unittest.TestLoader()
suite = loader.discover(start_dir='.', pattern='*Test.py')
runner = unittest.TextTestRunner(verbosity=5)
result = runner.run(suite)


Run all tests


At the command prompt, type:

python TestRun.py

(or run separate tests, for example like this: python -m unittest apiFunctionTest.py) and enjoy the results.

The disadvantages of this technique


The disadvantages of this technique include:

  • the relative complexity of the dll creation algorithm.
  • Possible problems related to type consistency and alignment issues in structures.
  • Debugging possible errors in the “adapter” file.
  • Great compilation time for individual dlls.
  • You must correctly and manually select compilation keys.
  • The need to install additional software.

findings


Of course, it’s good to use an IDE with built-in test support, but if not, then this technique can make life much easier. It is enough to spend time setting up a project testing system once. It should also be noted that it is possible to use the capabilities of parsing python to generate "live" documentation, and indeed the capabilities of python to work with program texts in C / C ++.

Link to the project archive .

Thank you for the attention.