Biometrics is being widely accepted for user authentication across the globe. Integration of biometrics in the daily life provokes the need to design secure authentication systems. This study proposes the use of outer ear images as a biometric modality. The comparable complexity between the human outer ear and face in terms of its uniqueness and permanence has increased interest in the use of ear as a biometric. However, similar to face recognition, it poses challenges of variation in illumination, contrast, rotation, scale and pose. Owing to the extensive work in the field of computer vision using convolutional neural networks (CNNs), its feasibility in the field of ear biometrics has been presented in this work. The proposed technique uses a CNN as a feature extractor and a support vector machine (SVM) for the classification task. The joint CNN-SVM framework is used for mapping ear images to random base-n codes. The codes are further hashed using the secure hash algorithm SHA-3 to generate secure ear templates. The feasibility of the proposed technique has been evaluated on annotated web ears dataset. This work demonstrates 12.52% average equal error rate without any image pre-processing, which shows that the proposed approach is promising in the field of secure ear biometrics