cv4logo - Computer Vision for Imagine Logo ------------------------------------------ version 16/02/2005 Introduction: cv4logo.ocx is a loadable library for Imagine Logo version 2. It contains some procedures and operations for image manipulation. It is based on the open-source OpenCV library from Intel. CVS snapshot installation: * checkout or update the latest cvs snapshot * make sure your system PATH environment variable contains the ..\modified.opencv.dll directory * make sure ocx is registered: Start->Run: Regsvr32.exe ..\robotika\cv4logo\cv4logo.ocx (or whatever path you have) * make sure Imagine is installed, and copy the haarcascades directory from cv4logo to Imagine directory Release package Installation: * unzip the package to some directory (let's call it $INSTALL) * make sure your system PATH environment variable contains the $INSTALL\modified.opencv.dll directory * register the cv4logo.ocx Start->Run: Regsvr32.exe $INSTALL\cv4logo.ocx * Copy the $INSTALL\haarcascades directory to your Imagine directory, which was installed beforehand Usage: This short usage manual is written for Imagine Logo, but the equivalent procedure and function calls can be made from any other application that loads this library as DLL. See example applications for more details. 1. CREATING AND SHOWING IMAGE ----------------------------- First create a new CV object, and name it for example cv: new "oleobject [comname cv4logo name cv] Now you can view the image from the camera in a separate window: cv'show or hide it (and later show it again): cv'hide To choose a different image source, you can either switch to another camera, file, or even URL: cv'source "cam 0 will use the first camera (default) cv'source "cam 1 will use another camera. To use a recorded avi file as a source for the frames to be processed, you can use: cv'source "avi "hello.avi For static image, the command cv'source "file "tomas.jpg will use the file tomas.jpg as the image source (note that the file formats BMP and PNG might work too), or cv'source "url "|http://www.bioinformatics.uwaterloo.ca/~tvinar/tomas.jpg| will download the image from the above address once and then use it as a source; however, if the image at given file or url location is changing all the time, you need to use different commands (this can be a case of a local or network webcam): cv'source "livefile "tomas.jpg or cv'source "liveurl "|http://adressa.no/template/img/webkamera/fullsize.jpg| The image will be normally updated 10-times per second (or slower, if the image source has lower frame rate), but you can change this value. For example, after this command is entered: cv'updaterate 30 the update rate will be changed to 30 fps. The value can be smaller than one, for example: cv'updaterate 0.1 will update every 10 seconds. Note that if the download or image capture or file load will not be fast enough, the actual frame-rate might be lower. The real achieved frame-rate (frames-per-second) can be determined by the operation: cv'getfps 2. DETECTING FACES OR OTHER PATTERNS ------------------------------------ You can try to detect faces in the camera view: cv'tracking "on Tracking can be turned off using cv'tracking "off And the detected faces will be shown by a red rectangle. You can get a list of the detected faces using command: make "faces first parse cv'getfaces And the command pr first :faces will then print the first face bounding rectangle found. You can change the tracking parameters file (for example to detect upper body part instead of front faces) using command: cv'trackingparameters "|haarcascades\haarcascade_upperbody.xml| If your camera is looking at some scene with polygons, you can detect them using the command: make "polygons cv'detectpoly "false pr :polygons If you specify "true as the argument, the detected polygons will be visualized in the image for a very short while. The detection uses several parameters that can be adjusted: cv'detectpolyparam threshold sensitivity minperimeter maxperimeter where threshold is an integer specifying the pixel color value that is used as a threshold between black and white areas (100 by default), sensitivity is a ratio relative to each shape perimeter that determines the detail level (0,02 by default), minperimeter and maxperimeter are the perimeter size boundaries between shapes detected and ignored. 3. CAMERA CALIBRATION, CLIPPING AND MOUSE SUPPORT ------------------------------------------------- Usual cameras that use to be connected to PC are not calibrated and the image is distorted. Each camera has different distortion and thus each camera has to be calibrated individually. You can do it from Logo using the command: make "param cv'calibration "true and showing the chessboard image (from pattern.pdf) in 4 different orientations in front of the camera so that all squares can be seen clearly by the camera. If you wish to get rid of the guiding dialog boxes, you can use "false as parameter and the calibration procedure will wait 5 seconds before each calibration frame is captured. If you prefer to have more accurate calibration and use higher number of frames, you can use number_of_frames as parameter, for instance; make "param cv'calibration 10 will capture and use 10 calibration frames. If you prefer to have the message boxes shown, put a negative sign in front of the parameter, for example: make "param cv'calibration -10 will capture and use 10 calibration frames and show the guiding message boxes. If the light conditions or chessboard position are inacurrate, you will need to start the calibration over again. The camera will start correcting the image distortion after: cv'calibrate :param (you can save the parameters to file, but note that you need to calibrate the camera for each distance separately, if you want to achieve more accurate image. Also note that :param has to be a _word_ in the same format as is returned by cv'calibration; that is, not a list). To stop using calibration (for example to return back and try to calibrate the camera again) use: cv'calibrate "off The calibration is tuned for printing the pattern.pdf file on A4 paper (square size 2.9 cm). However, you may obtain a little bit different square dimensions with your printer. You can set the square size using the following command before you compute the calibration parameters: cv'calibrationsquaresize cm For example, use: cv'calibrationsquaresize "3,0 if you printed on letter paper. After you calibrate your camera, it might be useful to remove some of the borders of your image. You can use the command: cv'clipview x1 y1 x2 y2 to show and work only with a clipped part of the original image. To cancel clipping, just specify large enough values, for instance cv'clipview 0 0 1000 1000 You can always ask cv4logo for the current frame size [width height] by: cv'framesize Simple mouse support is included as well, the operation cv'lastmouseclick returns a space-separated list of three numbers x y time_of_click, where time_of_click is the time in milliseconds since reboot, and the operation cv'mousepos returns the current mouse cursor position, if the mouse is over the cv4logo window. The upper-left image window corner has coordinates [0 0]. 4. STORING FRAMES TO FILES AND MEMORY ------------------------------------- You can save the current image or frame to a file (for example jpg, but other formats might work too) using command: cv'saveframe filename where filename is a word containing full path to the file name where the current frame should be saved If you want to work more with some frames without storing them to a file, you can store many frames into memory. The current frame (the same as is shown in the window) can be stored to memory using the operation cv'storeframe which returns a number - frame id using which you can retrieve it later by: cv'source "mem id you can erase a particular frame from the memory (a good idea if you do not need it anymore) using: cv'forgetframe id and you can erase all stored frames from the memory using: cv'forgetallframes you can also get the list of ids of all frames that are currently in the memory using: cv'getallstoredframes 5. USING CV4LOGO IN NETWORK --------------------------- It is possible to make the image frames available for other users in the local network or on the Internet who are also running cv4logo component. If you wish to do that, you need to enter the command: cv'frameserver "on And later possibly stop publishing the frames using cv'frameserver "off A remote user can than access the frames from such a server by specifying remote source: cv'source "remote "ip-address Note that the frameserver normally uses socket on port 1119. You can change it (if you for example want to run several frame servers on the same machine) using: cv'frameserverport port The library provides also other networking functionality that your application can use to communicate with other computers running cv4logo by sending/receiving text messages, or to download content from the Internet/Intranet. The message sending/receiving is the client/server kind, thus one of the communicating peers have to work as the server and accept connections from possibly multiple clients. The server listens for connections on some port using the server socket. All incoming connections are automatically accepted, but the application must explicitely pick them up. Each live connection has a unique integer identifier. The connection is established using the following procedure: 1. start listening on some port for incoming connections on the server side, for instance: make "servId cv'listen 12345 2. connect to the server from a client: make "connId cv'connect "192.168.145.62 12345 (you need to use the correct IP-address of the server - you can see it by running the ipconfig command from the cmd shell) 3. accept an incoming connection from client on server: make "client1 cv'accept :servId Now the connection is established and you can send or receive text messages in both directions using: cv'send connection_id message For example, to send a message from the server to client: cv'send :client1 "hello And to receive this message on client, use: pr cv'receive :connId The other direction would be: cv'send :connId "hiThere! pr cv'receive :client1 If another client connects to the same server, the connection has to be accepted by calling cv'accept :servId on the server. This procedure can be called at any time, and it will return 0 if there no new connection has been accepted lately. The send and receive procedures are blocking - that means they return only after a message was received or sent. If you only want to look if a new message has arrived, use cv'receivenb :client1 which will return empty word if no message arrived lately. If one of the peers closes the connection, no more messages can be sent. The status of the connection can be verified using command: pr cv'isconnected connectionId which returns word "true or "false. After the connection is not used anymore, it should be closed using: cv'close connectionId Finaly, the library can be used also to download files from the WWW. For example, the following command: cv'download "|http://opencv.sourceforge.net/| "cvhome.html will save the home page of the OpenCV to file cvhome.html (note that only the specified file will be saved; the linked images or documents will not be saved), or cv'download "|ftp://ftp.uu.net/graphics/jpeg/README| "jpeg.README will save the readme file from a jpeg software, if your computer is connected to the Internet. 6. LOCALISATION OF OBJECTS -------------------------- cv4logo supports tracking and localisation of multiple polygonal objects (for example robot). For each localised object you wish to trace you need to create an instance of localisator object: pr cv4logo'newlocalisator "type "parameter where type is the type of the object you want to trace (until now only type "robot is supported - denoting a circular black robot), and parameter is a parameter for the localisator (for the type "robot you can pass space-separated list of the following words, some of them followed by arguments (| denotes alternative, all words are optional - i.e. you can pass empty word): update_periodic T | update_on_posandheading | update_none guess_transform | copy_transform id_loc | transform sppx sppy expected_size d expected_pos x y snowman twosuns not_circular where the update* determine when the cv4logo will look in the image to try to localise your object. *transform determine how will cv4logo try to determine the transform between pixels and robot steps (either by guessing from first four robot locations, or by copying from another localised object - for example the same type of robot in the same scene, or given explicitely). The expected_size can determine the expected diameter of the robot. snowman localization is based on two white full circles beeing placed on top of black robot, smaller bouble in the front, larger bouble on the back. The center of the line connecting the centres of the two white circles is considered to be the reference point of the robot. two Suns localization is very simlar to snowman, but the robot should have two yellow circular stickers. This removes any confusion with other white polygons detected in case of snowman localization. use not_circular if your robot is not circular, for example, if it has a gripper. update_periodic is not implemented yet. In order to determine the size of your robot, you can use the following operation: cv'detectrobot expected-x expected_y expected-size where expected-* can be zero, if you do not care. It will locate a black circular polygon in the image that has size closest to expected-size and is closest to the given location. The operation newlocalisator returns a localisator identifier (integer), which you need to pass to all localisator operations in order to indicate which object are you tracing. The localisator can provide the current location of the object: cv'posandheading localisator_id In addition, the localisator can adjust the position of the object using complementary information (obtained for example from robot odometry). This information does not need to be scaled and does not need to have the same coordinate origin). You can send the updated location and heading information using procedure: cv'relposchange id-localisator pos-change where pos-change is a string of three double-precision floating-point numbers "dx dy dheading", which describe the relative position change of the object since the last call of relposchange. It is assumed that the position and heading is given with respect to the location of the object at previous call (including heading). For example, you can use this call to chain it with the analogical operation that returns the location and heading of the robot: cv'relposchange id_localisator r'pos 1 When you are done tracing some object, you should release the localisator by calling: cv'droplocalisator id-localisator The standard localisation procedure looks as follows: 1. create robot, create cv4logo 2. [optionally] detect the robot size: cv'detectrobotsize expected_size -> diameter 3. create localisator object for the robot by make "id_loc cv'newlocalisator "robot "update_on_posandheading\ guess_transform -> id_loc (only type "robot is supported, guess_transform will need the procedure in point 4, and update_on_posandheading will cause the cv4logo to use the image for localising the robot each time the posandheading is called) 4. initialize the localizator so that it can guess the current robot heading and location by the following procedure: r'pos 1 repeat 3 [ r'fd 1000 ignore r'wait 10000 r'rt 720 ignore r'wait 10000 wait 500 cv'relposchange :id_loc r'pos 1 ] r'fd 1000 ignore r'wait 10000 wait 500 cv'relposchange :id_loc r'pos 1 5. after each forward/backward movement command of the robot, call: cv'relposchange :id_loc r'pos 1 so that the cv4logo will update the robot location based on the robot's odometry If you are using speed commands, call the same periodically (say once per second) 6. the current location of the robot is available anytime by calling: cv'posandheading :id_loc -> "x y heading" 7. remember to drop the localisator when not used anymore: cv'droplocalisator id_loc Todo: * automatic setup of binary threshold value based on corner marks * detection of robot location and heading * better scene analysis * test Hough transform * compile as stand-alone, and static library * port to Linux :) * get image source from directx, not from video for windows :( Hints: * before using your camera, you can change its properties and video format using the program vidcap32.exe in your Windows directory (such as C:\WINNT) * multiple cv4logo objects can be created in logo to operate several cameras or other image sources Known issues: * changing the tracking after static image has been [down]loaded doesn't make a change in the image shown; the setting that is active during source selection decides tracking for static sources * loading gif and tif doesn't work due to the fact that OpenCV uses loaders that probably do fseek() and thus loading through pipe doesn't work; not sure if this will ever be fixed. neither saving gif does * in awkward situations, crashes might occur (please, report) Bugs: * after disconnecting network camera, cv'hide freezes Requirements: * Windows 98/2000 or newer (not tested on 98 though) * for use: dll libraries from _modified_ OpenCV beta 4, Friday, August 13, 2004 (included in the package) xml files for face or object recognition (included) * for development - OpenCV development environment with modifications (unless you plan to make more changes to OpenCV, you do not need the whole OpenCV package, all needed parts are included) - MS Visual studio 6 or .NET (might work with other compilers if you rebuild the workspace and project files) - libcurl 7.12.1 - Imagine v.2 (tested with build 341) History of changes: * new features in 2004-12-22: cv'calibrationsquaresize cm * variable number of frames for calibration cv'detectpolyparam threshold sensitivity minperimeter maxperimeter cv'clipview x1 y1 x2 y2 cv'frameserver "on cv'frameserverport port cv'source "remote ip-address cv'listen port cv'connect ip-address port cv'accept connection-id cv'send connection-id message cv'receive connection-id cv'receivenb connection-id cv'isconnected connection-id cv'close connection-id * new features in 2004-10-27: cv'detectpoly show cv'storeframe cv'source "mem id cv'forgetframe id cv'forgetallframes cv'getallstoredframes * new features in 2004-10-18: cv'source "cam/avi/file/url/livefile/liveurl "srcspec cv'updaterate x cv'getfps cv'download "url "filename cv'trackfaces renamed to cv'tracking cv'trackingparameters "xml-file Copyrights/Licences: * cv4logo: Copyright (c) 2004, DAI, FMPhI Comenius University, ppetrovic@acm.org * Imagine: Logotron Ltd. & EDI FMPhI * OpenCV: Copyright (C) 2000, 2001, Intel Corporation, all rights reserved. Third party copyrights are property of their respective owners. with modifications (C) 2004 DAI FMPhI Comenius University * libcurl: Copyright (c) 1996 - 2004, Daniel Stenberg, . and Copyright (c) 1998, 1999 Kungliga Tekniska Högskolan.