I have tried
this.ultraCalendarInfo1.Tasks.Remove(this.ultraGanttView1.ActiveTask);
but I get an Index out of range exception....
The lack of detail here makes it difficult to speculate but one possibility is that the active task at the time the Remove method is called is not a member of the root Tasks collection, i.e., is a descendant of some root level task. Tasks collections are nested, that is, the Tasks collection exposed by the UltraCalendarInfo object contains only the root-level tasks, and each of those tasks might have a non-empty Tasks collection.
Note that the Task class exposes a Parent property which represents its immediate parent; in the case where the task is a root-level task, this will be null, implying that the task belongs to the UltraCalendarInfo.Tasks collection. If the Parent property returns a non-null value, you can get the collection to which the task belongs via the Task.Parent.Tasks property.